Overview
Investigating logs is one of the most important tools for troubleshooting a Kubernetes deployment.However, there might be cases where a problem is occurring that is also causing problems with the apiserver pod.
In these cases you won't be able to retrieve logs by using "kubectl" commands.
This article will provide guidance on retrieving logs directly from an impacted node.
Resolution Steps
1. SSH into the affected node.2. Navigate to the log directory. By default, kube-system and kubeaddons pods should store their logs in /var/log/containers:
cd /var/log/containers ls -la
This ls command should list all of the log files in this directory, so you can identify the one you need to read.
3. Use a text viewer of your choice to investigate the logs on the system, or use SCP to copy the ones you need to a client machine.
In this example I'm using "less" to open my etcd log file since it's available on most linux systems by default. The filename will be different on your system so use the appropriate one from the "ls" output previously.
sudo less etcd-ip-x-x-x-x_kube-system_etcd-20ac994925b4b0701125430ffd9d1625d6136e5c8e3656aefba7b131f0ef61db.log