When the storage used by Loki to persists the data received from Fluent-Bit gets full (space or inodes being consumed), the Loki ingester will start complaining that there is not enough space to flush the data in memory and most probably will not pass its readiness check:
err="XMinioStorageFull: Storage backend has reached its minimum free disk threshold.
Please delete a few objects to proceed.\n\tstatus code: 507
When this happens, an alternative is to use the MinIO client to manually remove objects.
In this article we provide the steps to configure the MinIO client and delete objects from the Grafana Loki object storage backend.
The client has to be configured to query the service grafana-loki-minio-hl on port 9000. An option is to reach the service from a busybox pod where MinIO client is installed.
1. Create a pod with busybox image:
kubectl -n komamnder run -i --tty --rm debug --image=busybox --restart=Never -- sh
2. Download the MinIO client:
/ # wget https://dl.min.io/client/mc/release/linux-amd64/mc
/ # chmod +x mc
3. Configure alias to access the MinIO server endpoint. Credentials of a user with admin access to the MinIO object storage service is required. To access the grafana-loki-minio console, please refer to this article.
/ # ./mc alias set grafana_loki_minio http://grafana-loki-minio-hl.kommander.svc.cluster.local:9000
4. List the alias configured and confirm that the grafana-loki-minio storage is configured correctly:
/ # ./mc alias list
gcs
URL : https://storage.googleapis.com
AccessKey : YOUR-ACCESS-KEY-HERE
SecretKey : YOUR-SECRET-KEY-HERE
API : S3v2
Path : dns
grafana_loki_minio
URL : http://localhost:9000
AccessKey : <USERNAME>
SecretKey : <PASSWORD>
API : s3v4
Path : auto
local
URL : http://localhost:9000
AccessKey :
SecretKey :
API :
Path : auto
play
URL : https://play.min.io
AccessKey : Q3AM3UQ867SPQQA43P2F
SecretKey : zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG
API : S3v4
Path : auto
s3
URL : https://s3.amazonaws.com
AccessKey : YOUR-ACCESS-KEY-HERE
SecretKey : YOUR-SECRET-KEY-HERE
API : S3v4
Path : dns
5. Test whether the client can list the objects or not:
/ # ./mc ls grafana_loki_minio/loki/fake
6. Before deleting objects is always a good idea to perform a dry-run test to confirm that desired objects will be deleted:
/ # ./mc rm --dry-run --recursive --force --older-than 60d0h0s grafana_loki_minio/loki/fake
7. Objects deletion (in this case objects older than 60 days will be deleted):
/ # ./mc rm --recursive --force --older-than 60d0h0s grafana_loki_minio/loki/fake
A second option is to reach the service from outside the cluster, to do it that way, a port-forward can be configured by executing the command:
kubectl port-forward -n kommander svc/grafana-loki-minio-hl 9000:9000
Then install the MinIO client locally as described previously in this article, and configure the alias to reach the grafana-loki-minio object storage service via http://localhost:9000:
./mc alias set grafana_loki_minio http://localhost:9000