Use a specific storageClass for Rook-ceph

Background

Rook-ceph requires a storage with volumeMode: Block for it's OSD's (Object Storage Daemons).

For Cloud based infrastructure, there is usually no issue, as the default storageClass is capable of creating the required PV's.

But for on-prem/preprovisioned clusters, which uses localvolumeprovisioner as the default storageClass, this can be a challenge. Please refer to the KB article for adding raw storage and configuring cephCluster to use this raw storage - https://support.d2iq.com/hc/en-us/articles/12618140300948-Rook-Ceph-Installation-Errors-for-On-Premise

Although, for those clusters that has a storage solution deployed, for example Portworkx. There is usually multiple storageClass available. And one of these storageClasses support's volumeMode: Block. For this cases, we prefer to use this storageClass as a PVC mode, rather than provisioning raw storage on the nodes.

Solution

If unsure, please consult your storage provider, if their solution support's Block volumeMode. And if there is a need to specify which storageClass would be used. We have to 

1. Create a configMap that would specify the storageClass name to be used by Ceph.
2. Modify the AppDeployment to use the created configMap.

Create the configMap specifying the <storageClass_name> to be used. Please see the following example

apiVersion: v1
kind: ConfigMap
metadata:
  name: rook-ceph-cluster-for-portworx
  namespace: kommander
data:
  values.yaml: |
    ---
    toolbox:
      enabled: true

    # All values below are taken from the CephCluster CRD
    cephClusterSpec:

      storage:
        storageClassDeviceSets:
          - name: rook-ceph-osd-set1
            count: 4
            # This should be set to false if your StorageClass does not support porting PVs from one node to another (E.g.: ebs-sc does support this, but local provisioner does not).
            portable: true
            encrypted: false
            placement:
              topologySpreadConstraints:
              - maxSkew: 1
                topologyKey: topology.kubernetes.io/zone # The nodes in the same rack have the same topology.kubernetes.io/zone label.
                whenUnsatisfiable: ScheduleAnyway
                labelSelector:
                  matchExpressions:
                    - key: app
                      operator: In
                      values:
                        - rook-ceph-osd
                        - rook-ceph-osd-prepare
              - maxSkew: 1
                topologyKey: kubernetes.io/hostname
                whenUnsatisfiable: ScheduleAnyway
                labelSelector:
                  matchExpressions:
                    - key: app
                      operator: In
                      values:
                        - rook-ceph-osd
                        - rook-ceph-osd-prepare
            volumeClaimTemplates:
              # If there are some faster devices and some slower devices, it is more efficient to use
              # separate metadata, wal, and data devices.
              # Refer https://rook.io/docs/rook/v1.10/CRDs/Cluster/pvc-cluster/#dedicated-metadata-and-wal-device-for-osd-on-pvc
              - metadata:
                  name: data
                spec:
                  resources:
                    requests:
                      storage: 40Gi
                  # Use the default storage class configured in cluster.
                  # not setting storageClass to let it fall to environment default
                  # OSD Requires Block storage.
                  storageClassName: <storageClass_name>
                  volumeMode: Block
                  accessModes:
                    - ReadWriteOnce

Modify the Rook-Ceph-Cluster AppDeployment

kubectl edit appdeployments -n kommander rook-ceph-cluster

under spec include the configOverrides.name, example:

spec:
  appRef:
    kind: ClusterApp
    name: rook-ceph-cluster-1.10.3
  configOverrides:
    name: rook-ceph-cluster-for-portworx

By updating the AppDeployment, the configuration will not be overwritten on future updates.

The helmrelease should reconcile automatically, to apply the new changes. But for new deployment, and the helmrelease has already exhausted the retries due to failure. Please force the helmrelease reconciliation with the following commands:

kubectl -n kommander patch helmrelease rook-ceph-cluster --type='json' -p='[{"op": "replace", "path": "/spec/suspend", "value": true}]'
kubectl -n kommander patch helmrelease rook-ceph-cluster --type='json' -p='[{"op": "replace", "path": "/spec/suspend", "value": false}]'