How to renew the TLS certificates used by Prometheus to scrape etcd metrics – D2iQ

Issue Description
To fetch metrics out of etcd, Prometheus uses TLS certificates stored in the etcd-certs secret in the kubeaddons namespace.

When the TLS certificate expires, Prometheus cannot collect information from etcd due to the invalid TLS certificate and grafana etcd dashboard will be affected. To confirm whether this issue has been encountered, users could check etcd logs for events like this:

2021-02-08 07:09:24.080338 I | embed: rejected connection from "10.156.240.229:47718" (error "tls: failed to verify client's certificate: x509: certificate has expired or is not yet valid", ServerName "")

Solution
To fix the issue, etcd certificates must be renewed and the secret etcd-certs updated as described in the following procedure:

1. SSH into a control plane node and back up the current etcd certs and secret:

kubectl --kubeconfig /etc/kubernetes/admin.conf get secret etcd-certs -n kubeaddons -o yaml > ~/etcd-secret-backup.yaml && cp -r /etc/kubernetes/pki ~

2. Check the etcd certs:

kubeadm alpha certs check-expiration

Note: when Konvoy is upgraded, etcd TLS certificates are renewed, however, the secret (etcd-certs) is not updated with the new certificate. If the etcd certificates are not expired, the operator can skip step 3 and go to step 4.

3. Renew etcd certificate:

kubeadm alpha certs renew etcd-server

4. Update the secret that stores the TLS certificate used by Prometheus to scrape etcd:

set -o pipefail && kubectl --kubeconfig /etc/kubernetes/admin.conf create secret generic etcd-certs --namespace kubeaddons --from-file=/etc/kubernetes/pki/etcd/ca.crt --from-file=/etc/kubernetes/pki/etcd/server.crt --from-file=/etc/kubernetes/pki/etcd/server.key --dry-run=client -o yaml | kubectl --kubeconfig /etc/kubernetes/admin.conf apply -f -

5. Check that the cert has the expected output (expiration):

kubectl --kubeconfig /etc/kubernetes/admin.conf get secret -n kubeaddons etcd-certs -o jsonpath='{.data.server\.crt}' | base64 --decode | openssl x509 -text -noout

6. Update the Prometheus pods with the new cert:

kubectl --kubeconfig /etc/kubernetes/admin.conf rollout restart sts/prometheus-prometheus-kubeaddons-prom-prometheus -n kubeaddons

7. Ensure the cert is updated inside of Prometheus container:

kubectl exec -it prometheus-prometheus-kubeaddons-prom-prometheus-0 -n kubeaddons -- /bin/sh -c 'cat /etc/prometheus/secrets/etcd-certs/server.crt' 2> /dev/null | openssl x509 -text -noout