Skip to main content

Grafana-logging logs are not up to date, with fluentbit errors

Alex Christopher Ramos
  • Updated

Issue

Application logs in Grafana-logging is not up to date and when observing the logs of FluentBit, there error lines similar to:

kubectl logs -l=app.kubernetes.io/name=fluentbit -n kommander --max-log-requests=7 -f
[2023/01/04 07:43:17] [error] [net] TCP connection failed: logging-operator-logging-fluentd.kommander.svc.cluster.local:24240 (Connection refused)
[2023/01/04 07:43:17] [error] [output:forward:forward.0] no upstream connections available
[2023/01/04 07:43:17] [ warn] [engine] failed to flush chunk '1-1672818196.694387935.flb', retry in 6 seconds: task_id=2, input=tail.0 > output=forward.0 (out_id=0)
[2023/01/04 07:43:17] [error] [net] TCP connection failed: logging-operator-logging-fluentd.kommander.svc.cluster.local:24240 (Connection refused)

Solution

First attempt to restart the fluentbit daemonset by running the command below:

kubectl rollout restart ds logging-operator-logging-fluentbit -n kommander

If the issue still occurs, please increase the memory resource/request limit of the FluentD deployment, as mentioned in this guide.