|Synopsis:||When upgrading to Konvoy 1.7.3 or Konvoy 1.8.1, containerd configuration files are not correctly copied over leading to unexpected behavior. New installations are not impacted.|
|Affected Products & Versions||
Konvoy 1.7.3, Konvoy 1.8.1
Konvoy 1.7.3 and 1.8.1 included an upgrade of containerd from 1.3.x to 1.4.6; however, due to an upstream issue with the upgrade procedure of containerd, custom configuration files were not correctly copied over. This results in a cluster being left with no /etc/containerd/config.toml, only the rpmsave/rpmnew files, and thus any custom registry locations or credentials will no longer be in effect. You will see this issue on an impacted instance after the first restart of containerd after completing the upgrade. Note that this issue only impacts upgrades from an older version of Konvoy; a new install of the impacted Konvoy versions does not experience the issue.
A fix for this issue is included in the next patch release of Konvoy (Konvoy 1.7.4, 1.8.2) via containerd 1.4.7.
Please do not upgrade to Konvoy 1.7.3 or 1.8.1; instead upgrade directly to Konvoy 1.7.4 or Konvoy 1.8.2 . If you have already upgraded, please follow the below steps to manually fix your instances. An Ansible playbook is included at the end of this advisory that can be used to restore the missing file.
- Move /etc/containerd/config.toml.rpmsave to /etc/containerd/config.toml via cp /etc/containerd/config.toml.rpmsave /etc/containerd/config.toml
- Restart containerd with sudo systemctl restart containerd
You can use the following playbook to restore the missing config.toml files on impacted clusters. Save the text below as fix_containerd.yaml in the same directory as your cluster.yaml file, and then run the command
konvoy run playbook fix_containerd.yaml -y
--- - hosts: node control-plane name: "Restore containerd config.toml" tasks: - name: check if /etc/containerd/config.toml.rpmsave exists stat: path: /etc/containerd/config.toml.rpmsave register: containerd_config_toml_rpmsave - name: check if /etc/containerd/config.toml exists stat: path: /etc/containerd/config.toml register: containerd_config_toml - name: Copy config.toml.rpmsave to config.toml iff there is no existing config.toml become: yes copy: src: /etc/containerd/config.toml.rpmsave dest: /etc/containerd/config.toml remote_src: yes owner: root group: root mode: '0644' when: - containerd_config_toml_rpmsave.stat.exists - not containerd_config_toml.stat.exists