How to find kube-apiserver audit logs in Konvoy

Overview

In some cases, it might be necessary to see which requests are being made through your Konvoy cluster's kube-apiserver.

The audit logs are stored in a different location than the stdout pod logs of kube-apiserver, so this article will show you how to find the location of them.

How to

First, retrieve the pod IDs of the kube-apiserver pods along with the hosts each one is running on:

kubectl get pods -n kube-system -o wide 

The output should resemble the following (other pods removed from this example):

NAME                              READY   STATUS    RESTARTS   AGE   IP                NODE       NOMINATED NODE   READINESS GATES
kube-apiserver-node01             1/1     Running   1          22m   10.X.X.X          node01                
kube-apiserver-node02             1/1     Running   0          22m   10.X.X.X          node02                
kube-apiserver-node03             1/1     Running   2          21m   10.X.X.X          node03                

To find the location of the audit logs of these pods, get the yaml configuration by referencing one of the pod IDs you just got. In this example, I would use:

kubectl get pod -n kube-system kube-apiserver-node01 -o yaml 

In the output you receive, there will be two relevant sections. First, the configuration that describes the file location of the audit log inside the kube-apiserver container:

    - --audit-log-path=/var/log/audit/kube-apiserver-audit.log

The other will be the volume mounts that are configured by default to describe the location of the same file on the host machine where the kube-apiserver pod is running:

  volumes:
  - hostPath:
      path: /var/log/kubernetes/audit
      type: ""
    name: audit-logs

To retrieve the logs directly from the container, I can use kubectl exec:

kubectl exec -n kube-system kube-apiserver-node01 cat /var/log/audit/kube-apiserver-audit.log > /tmp/audit.log

This will execute cat directly in the container to print the contents of the audit log and write the output to /tmp/audit.log on my local machine.

If you wish to retrieve the contents on the host machine instead of running a command directly within the container, you can ssh into the host machine and retrieve them from the hostPath location as described above; in this case, /var/log/kubernetes/audit.