Overview
A fundamental requirement of stateful distributed systems is that time must be synchronized between all nodes in the system so that transactions generated and processed by stateful components can be properly ordered.
Because etcd, the key-value store used by Kubernetes, is a stateful distributed component, a Kubernetes cluster requires robust time synchronization between the nodes. We recommend the use of NTP or Chrony to synchronize the clocks on nodes with an upstream time source.
DKP by default installs and configures Chrony to use default upstream time servers. If you have customized the configuration, or the default timeservers are not reachable, then time on the your cluster nodes may become unsynchronized.
To customize the config, you can edit the KubeadmConfigTemplate.spec.template.spec.ntp field and specify other servers.
To verify that a host is configured to use NTP/Chrony, and time is synchronized, use the timedatectl command:
$ timedatectl
Local time: Mon 2021-11-29 22:12:26 UTC
Universal time: Mon 2021-11-29 22:12:26 UTC
RTC time: Mon 2021-11-29 22:12:26
Time zone: UTC (UTC, +0000)
NTP enabled: yes
NTP synchronized: yes
RTC in local TZ: no
DST active: n/a
You should verify that the NTP enabled and NTP synchronized are both `yes`.
If they are not, then please verify that either NTP or Chrony are installed and configured properly.
For more information on time synchronization, please consult the following resources:
http://www.ntp.org/ntpfaq/NTP-a-faq.htm
https://chrony.tuxfamily.org/faq.html