Skip to main content

Removing Rook Ceph and Configuring Grafana Loki to use an external S3 bucket

Mark Kamsika
  • Updated
 
A user may wish to remove Rook Ceph for performance reasons. Grafana-Loki and Velero can be configured to use an external S3 bucket removing the dependency on Rook Ceph/Rook Ceph Cluster.
 
This article covers both removing Rook Ceph/Rook Ceph Cluster and configuring Grafana-Loki and Velero to use an S3 bucket.
 
Solution
1. Remove metadata.finalizers from the following resources:
  • cephobjectstore dkp-object-store
  • objectBucketClaims dkp-loki dkp-velero
  • objectBuckets obc-kommander-dkp-loki obc-kommander-dkp-velero
  • secrets dkp-loki dkp-velero
Use the command below which opens the vi editor, edit each file removing the finalizers and save.
kubectl -n kommander  edit <resource> <resource name>

2. Remove the cephObjectStore, objectBuckets and objectBucketClaims resources used by Grafana Loki and Velero:

kubectl -n kommander delete cephobjectstore dkp-object-store
kubectl -n kommander delete obc dkp-loki dkp-velero
kubectl delete ob obc-kommander-dkp-loki obc-kommander-dkp-velero

3. Uninstall Velero, Grafana Loki, Rook Ceph and Rook Ceph cluster via the kommander GUI.

4. Delete the existing dkp-loki and dkp-velero secrets:
kubectl -n kommander delete secret dkp-loki dkp-velero

5. Create new dkp-loki and dkp-velero secrets with the required AWS access ID and Secret:

kubectl create secret generic dkp-loki \
                      --from-literal=AWS_ACCESS_KEY_ID=<key>\
                      --from-literal=AWS_SECRET_ACCESS_KEY=<secret>
kubectl create secret generic dkp-velero \
                      --from-literal=AWS_ACCESS_KEY_ID=<key>\
                      --from-literal=AWS_SECRET_ACCESS_KEY=<secret>

6. Create an installation file for grafana loki and velero with the following overrides:

apiVersion: config.kommander.mesosphere.io/v1alpha1
kind: Installation
apps:
  grafana-loki:
    enabled: true
    values: |
      loki:
        structuredConfig:
          storage_config:
            aws:
              s3: https://s3.amazonaws.com
              bucketnames: loki-bucket
              endpoint: s3.eu-west-2.amazonaws.com
              region: eu-west-2
              insecure: false
              sse_encryption: false
  velero:
    enabled: true
    values: |
      configuration:
        provider: "aws"
        backupStorageLocation:
          bucket: velero-bucket
          config:
            region: eu-west-2
            s3Url: https://s3.eu-west-2.amazonaws.com
      credentials:
        extraSecretRef: dkp-velero

Please note that the s3 key must be present in the Grafana-Loki configuration as its needed to override the default values provided by DKP which are:

storage_config:
  aws:
    s3: "http://rook-ceph-rgw-dkp-object-store.kommander.svc:80/dkp-loki"
    s3forcepathstyle: true

You must not specify accessKeyId or secretAccessKey. Specifying these keys will result in the Grafana Loki pods failing with an error similar to the one below:

failed parsing config: /etc/loki/config/config.yaml: yaml: unmarshal errors:
line 89: field accessKeyId not found in type aws.StorageConfig
line 95: field secretAccessKey not found in type aws.StorageConfig

You cannot use the same bucket for grafana-loki and velero.

6. Apply the installation file:

dkp install kommander --installer-config values.yaml