How to read kube-apiserver audit logs
Overview
If you follow the article linked here, it can help you with retrieving the kube-apiserver audit logs: https://support.d2iq.com/s/article/How-to-find-kube-apiserver-audit-logs-in-Konvoy.
After you've retrieved them, you can use this guide to walk you through an example of an audit log line and the meaning of the most relevant bits of information.
Solution
Here is an example log line that was generated after running "kubectl get nodes" in a DKP cluster:
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Request","auditID":"a7346465-5c58-49c5-8e3c-f83f658fb720","stage":"ResponseComplete","requestURI":"/api/v1/nodes?limit=500","verb":"list","user":{"username":"kubernetes-admin","groups":["system:masters","system:authenticated"]},"sourceIPs":["10.0.66.254"],"userAgent":"kubectl/v1.16.11 (linux/amd64) kubernetes/436254b","objectRef":{"resource":"nodes","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2020-08-06T22:42:50.370237Z","stageTimestamp":"2020-08-06T22:42:50.372383Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":""}}
You might notice that the kube-apiserver audit log lines are in JSON format. It can make them much easier to read if you run them through a tool to add newlines and indentation. Here is what the same line looks like when run through one of them:
{ "kind": "Event", "apiVersion": "audit.k8s.io/v1", "level": "Request", "auditID": "a7346465-5c58-49c5-8e3c-f83f658fb720", "stage": "ResponseComplete", "requestURI": "/api/v1/nodes?limit=500", "verb": "list", "user": { "username": "kubernetes-admin", "groups": [ "system:masters", "system:authenticated" ] }, "sourceIPs": [ "10.0.1.2" ], "userAgent": "kubectl/v1.16.11 (linux/amd64) kubernetes/436254b", "objectRef": { "resource": "nodes", "apiVersion": "v1" }, "responseStatus": { "metadata": {}, "code": 200 }, "requestReceivedTimestamp": "2020-08-06T22:42:50.370237Z", "stageTimestamp": "2020-08-06T22:42:50.372383Z", "annotations": { "authorization.k8s.io/decision": "allow", "authorization.k8s.io/reason": "" } }
Now we can see that this log entry makes a lot more sense! We'll describe the most relevant parts below.kind
, apiVersion
: These are values that every Kubernetes object need to have, to describe the type of object and which API version that object belongs to. This is sort of like referencing a library in a programming language. For the purpose of reading this log, however, we can just assume that the "kind" of every entry is going to be "Event".level
: This is the level of the log entry. Your audit logs can be configured to only log certain levels. This entry is a Request
level entry, but other levels include Metadata
and RequestResponse
.auditID
: This is the ID of the audit log entry. Each one is unique.stage
: This describes the part of the request that the log was generated during. There might be different bits of information that can be gathered at different parts of the transaction, so this field might show that the log was generated during the stages of RequestReceived
, ResponseStarted
, ResponseComplete
, or Panic
.requestURI
: This is the Universal Resource Identifier that the request was made to. This is the sub-path under the objectRef.resource
field, described in more detail below.verb
: This describes the permission verb that was required in order to access the resource that is being requested. In this case, my kubectl get nodes
command only requested a resource that requires list
permissions on nodes. See the Kubernetes documentation for more about request verbs: https://kubernetes.io/docs/reference/access-authn-authz/authorization/#determine-the-request-verbuser
: This describes the user that made the request. In this case, my kubectl client was using the "kubernetes-admin" user. groups
describes the ClusterRoleBindings that are assigned to that user which are then checked to make sure they are allowed to access the resource.sourceIPs
: This is the source of the request. In this example the source IP is the internal load balancer address that my kubectl client is pointed to. I would be able to compare access records in AWS if I needed to track the original source of the request to that.userAgent
: This refers to the specific method of access that was used to make the request. In this case it shows the specific version and build of kubectl that I was using. This field will not be present if a direct API call was made. It might show information about other clients if the request came from them, like a specific web browser that accessed a dashboard or the Kubernetes client of a specific language like Java.objectRef
: This field describes the object that was requested. kubectl commands are all simple HTTP requests to the API server, so in this case, my "kubectl get nodes" command made a request to the "nodes" resource, using API version "v1", to the endpoint "/api/v1/nodes?limit=500" (which we got from the requestURI
field above).responseStatus
: This is the HTTP response code that the kube-apiserver returned to the client that made the request. In this case it's a 200, which is a successful request. The metadata
field might contain more information like an error message if the status shows another code.requestReceivedTimestamp
: The time that the request was received by the kube-apiserver.stageTimestamp
: The time that the request reached the current stage, in this case ResponseComplete
. If the log entry is in a different stage, like ResponseStarted
, the timestamp would reflect when the ResponseStarted
stage was logged.annotations
: Miscellaneous metadata that is attached to any Kubernetes object. In this case, it shows the decision of whether the request was allowed or denied, and the reason if it was denied.
For more information about kube-apiserver audit logs, please feel free to consult the Kubernetes documentation: https://kubernetes.io/docs/tasks/debug-application-cluster/audit/.