When troubleshooting issues related to the grafana-loki app-deployment in Kommander, in addition to checking the pod logs, in certain scenarios it is also useful to query Loki components endpoints that provide information about the status of the component, configurational details or just gracefully shutdown the components.
kubectl run -i --tty --rm loki-endpoints --image=ubuntu --restart=Never -- sh
The ingester /ready endpoint is useful when there are issues with this component. Below is an example of how the response looks like when the ingester is up and healthy:
curl http://grafana-loki-loki-distributed-ingester.kommander.svc.cluster.local:3100/ready
ready
When the Loki ingester is not ready, the response may contain that some components are not running:
curl http://grafana-loki-loki-distributed-ingester.kommander.svc.cluster.local:3100/ready
Some services are not Running:
Terminated:1
Running:5
Or a connection refuse because the backend pod grafana-loki-loki-distributed-ingester-0 is not running and therefore cannot accept any connections:
curl http://grafana-loki-loki-distributed-ingester.kommander.svc.cluster.local:3100/ready
curl: (7) Failed to connect to grafana-loki-loki-distributed-ingester.kommander.svc.cluster.local port 3100 after 1003 ms: Connection refused
The /ready endpoint is also exposed by other components. For a complete list please refer to grafana-loki documentation.
Another useful endpoint is /config especially when applying changes to the Loki ingester configuration, as querying the endpoint allows to confirm whether the changes were applied or not:
curl http://grafana-loki-loki-distributed-ingester.kommander.svc.cluster.local:3100/config
Last but not least is the flush_shutdown, flush, and shutdown endpoints to perform the actions suggested by their names. flush_shutdown in particular is useful to force the Loki ingester to persist data (chunks) in memory on shutdown.
curl -ILv -XPOST http://grafana-loki-loki-distributed-ingester.kommander.svc.cluster.local:3100/ingester/flush_shutdown
For additional details about the endpoints exposed by each component please refer to the grafana-loki documentation.