What is conntrack?
https://manpages.ubuntu.com/manpages/bionic/man8/conntrack.8.html#:~:text=Using%20conntrack%20%2C%20you%20can%20dump,line)%20per%20newly%20established%20connection.
Conntrack provides a full featured userspace interface to the netfilter connection
tracking system that is intended to replace the old /proc/net/ip_conntrack interface. This
tool can be used to search, list, inspect and maintain the connection tracking subsystem
of the Linux kernel.
Why am I getting alerts about conntrack?
As the load on your Kubernetes cluster increases, you will have more and more connections that are tracked utilizing conntrack. If you have too many connections tracked, conntrack will start dropping connections which is generally bad.
Can I just disable conntrack?
No, Kubernetes requires conntrack to function properly.
How do I resolve the conntrack_max alerts?
Normally, this is a value that you would edit at the OS level using tools like sysctl. However, in Kubernetes, the conntrack values are managed by kube-proxy on each node.
To change the value, you will need to change the "--conntrack-max-per-core" option directly in kube-proxy:
https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/#options
To make this change in a DKP cluster, the best way is to edit the "kube-proxy" configmap in the "kube-system" namespace:
kubectl edit configmap -n kube-system kube-proxy
In the configmap, you will see a block that looks like this:
conntrack:
maxPerCore: null
min: null
tcpCloseWaitTimeout: null
tcpEstablishedTimeout: null
You can change the "maxPerCore: null" value to an appropriate value for your environment, e.g.:
conntrack:
maxPerCore: 65535
min: null
tcpCloseWaitTimeout: null
tcpEstablishedTimeout: null
Bear in mind that this is a per core value, and it defaults to 32768. If your nodes have 8 cores, then the actual "net.nf_conntrack_max" on that node will be set to 262144. Be mindful of how many CPU cores your nodes have so you can do the necessary math to arrive at your desired conntrack value.
After increasing this value, you will see slightly higher memory usage on your nodes due to the increased network connection tracking but this should not greatly impact your resource utilization on the Kubernetes nodes.