Some Kommander 2.X users have encountered that Loki drops log lines due to out-of-order timestamps. Below there is an example on how this type of events in the Loki ingester log:
level=warn ts=2022-08-24T15:05:48.007656989Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=213.639µs err="rpc error: code = Code(400) desc = entry with timestamp 2022-08-24 14:29:20 +0000 UTC ignored, reason: 'entry out of order' for stream: {log_source=\"kubernetes_host_kernel\"},\n entry with timestamp 2022-08-24 14:29:20 +0000 UTC ignored, reason: 'entry out of order' for stream: {log_source=\"kubernetes_host_kernel\"},\nentry with timestamp 2022-08-24 14:29:20 +0000 UTC ignored, reason: 'entry out of order' for stream: {log_source=\"kubernetes_host_kernel\"},\n entry with timestamp 2022-08-24
This issue affects older Loki versions (older than 2.4 which is included in DKP 2.2.2 and below) where it was required that log lines arrived in order by timestamp. This requirement was lifted in v2.4. For additional details please refer to this blog.
In Kommander 2.3, Loki has been upgraded to v2.5 so the entry out-of-order issue should not be encountered anymore in that version.
If for some reason upgrading to Kommander 2.3 is not possible and the user is in need of upgrading the Loki version, the user could modify the following configmap in the gitea repository:
dkp/kommander/git/kommander/kommander/raw/branch/main/services/grafana-loki/0.33.2/defaults/cm.yaml
By adding the image registry, repository and tag to the aforementioned configmap:
In addition, the "split_queries_by_interval" has to be included in the query_range section, or Loki will get upset about it: