How to find kube-apiserver audit logs in Konvoy
In some cases, it might be necessary to see which requests are being made through your Konvoy cluster's kube-apiserver.
The audit logs are stored in a different location than the stdout pod logs of kube-apiserver, so this article will show you how to find the location of them.
First, retrieve the pod IDs of the kube-apiserver pods along with the hosts each one is running on:
kubectl get pods -n kube-system -o wide
The output should resemble the following (other pods removed from this example):
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-apiserver-node01 1/1 Running 1 22m 10.X.X.X node01 kube-apiserver-node02 1/1 Running 0 22m 10.X.X.X node02 kube-apiserver-node03 1/1 Running 2 21m 10.X.X.X node03
To find the location of the audit logs of these pods, get the yaml configuration by referencing one of the pod IDs you just got. In this example, I would use:
kubectl get pod -n kube-system kube-apiserver-node01 -o yaml
In the output you receive, there will be two relevant sections. First, the configuration that describes the file location of the audit log inside the kube-apiserver container:
The other will be the volume mounts that are configured by default to describe the location of the same file on the host machine where the kube-apiserver pod is running:
volumes: - hostPath: path: /var/log/kubernetes/audit type: "" name: audit-logs
To retrieve the logs directly from the container, I can use kubectl exec:
kubectl exec -n kube-system kube-apiserver-node01 cat /var/log/audit/kube-apiserver-audit.log > /tmp/audit.log
This will execute
cat directly in the container to print the contents of the audit log and write the output to /tmp/audit.log on my local machine.
If you wish to retrieve the contents on the host machine instead of running a command directly within the container, you can ssh into the host machine and retrieve them from the hostPath location as described above; in this case, /var/log/kubernetes/audit.