Problem

In some cases, you may see that Chartmuseum is failing to deploy due to an issue creating its persistent volume. When this happens, you will see an error similar to the one below when describing the pod:

Warning FailedScheduling 8m8s (x132 over 22h) default-scheduler running PreBind plugin 
"VolumeBinding": binding volumes: timed out waiting for the condition

Solution

When you face this error you may be able to gain more information by viewing the EBS-Driver pods logging:

E0407 13:12:20.933969 1 controller.go:981] error syncing claim ": failed to provision volume with StorageClass "ebs-sc": rpc error: code = DeadlineExceeded desc = context deadline exceeded
I0407 13:12:20.933995 1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"kommander", Name:"chartmuseum", UID:"", APIVersion:"v1", ResourceVersion:"12915", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "ebs-sc": rpc error: code = DeadlineExceeded desc = context deadline exceeded

If you cannot identify the root cause by viewing the ebs-driver pods logging, the next best step is to SSH onto one of your nodes and validate that you can create the EBS volume using the AWS CLI. This can be done with the below command:

aws ec2 create-volume --volume-type <volume in the ebs storage class> --size 80 --availability-zone <AZ> --region <region>

If this command fails, then you will need to validate that your attached IAM role has proper permissions to create EBS volumes. It is important to note that by default, the EBS-Driver will attempt to create unencrypted volumes. If your organization has a service control policy that enforces encryption on EBS volumes, you will need to edit the ebs storage class to contain the following:

spec: 
  encrypted: true
  kmsKeyId: <ARN FOR KEY>
...

Please note that the storage class cannot be edited so you will need to backup the configuration with kubectl get sc ebs-sc -o yaml > ebs.yaml add in the encryption data, delete the storage class from the cluster, then re-apply the edited YAML. After doing so, you may re-run the Kommmander installation command.