How to collect logs from the apps that do not write to stdout or stderr

Problem

Some applications do not write logs to standard outputs. It is problematic to collect logs from such applications in Kubernetes clusters as kubelets clean up sandboxes of the stopped containers quite rapidly.

Solution

If, for some reason, it's not possible to configure the app to write logs to stdout, you can try adding a sidecar container (or containers, if the app is writing to multiple logs) that reads from the file and redirects to stdout. Then you can deal with the logs in one of the standard ways -- using log aggregators or copying logs from the /var/log/pods/ directory of the correspondent node.

Example

The following app consists of two containers. The first container writes its log to a filesystem. The second one constantly reads from that log file and redirects it to stdout.

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: twelve-factor-app
  name: twelve-factor-app
spec:
  volumes:
  - name: logs
    emptyDir: {}
    
  containers:
  - name: file-logger
    image: busybox
    args:
    - sh
    - -c
    - while true; do date >> /pod_logs/local.log; sleep 2; done
    
    volumeMounts:
    - name: logs
      mountPath: /pod_logs

  - name: log-reader
    image: busybox
    args:
    - sh
    - -c
    - tail -f /pod_logs/local.log
    
    volumeMounts:
    - name: logs
      mountPath: /pod_logs
  dnsPolicy: ClusterFirst
  restartPolicy: Always

You can test the solution with the following command:

kubectl logs twelve-factor-app logger-reader -f