Customer Advisory
| Advisory ID: | D2IQ-2021-0004 |
|---|---|
| Severity: | High |
| Synopsis: | When upgrading to Konvoy 1.7.3 or Konvoy 1.8.1, containerd configuration files are not correctly copied over leading to unexpected behavior. New installations are not impacted. |
| Affected Products & Versions |
Konvoy 1.7.3, Konvoy 1.8.1 |
| Issue date: | 07-27-2021 |
| Updated on: | 07-27-2021 |
Problem Description
Konvoy 1.7.3 and 1.8.1 included an upgrade of containerd from 1.3.x to 1.4.6; however, due to an upstream issue with the upgrade procedure of containerd, custom configuration files were not correctly copied over. This results in a cluster being left with no /etc/containerd/config.toml, only the rpmsave/rpmnew files, and thus any custom registry locations or credentials will no longer be in effect. You will see this issue on an impacted instance after the first restart of containerd after completing the upgrade. Note that this issue only impacts upgrades from an older version of Konvoy; a new install of the impacted Konvoy versions does not experience the issue.
Workaround/Solution
A fix for this issue is included in the next patch release of Konvoy (Konvoy 1.7.4, 1.8.2) via containerd 1.4.7.
Please do not upgrade to Konvoy 1.7.3 or 1.8.1; instead upgrade directly to Konvoy 1.7.4 or Konvoy 1.8.2 . If you have already upgraded, please follow the below steps to manually fix your instances. An Ansible playbook is included at the end of this advisory that can be used to restore the missing file.
- Move /etc/containerd/config.toml.rpmsave to /etc/containerd/config.toml via cp /etc/containerd/config.toml.rpmsave /etc/containerd/config.toml
- Restart containerd with sudo systemctl restart containerd
Ansible Playbook
You can use the following playbook to restore the missing config.toml files on impacted clusters. Save the text below as fix_containerd.yaml in the same directory as your cluster.yaml file, and then run the command
konvoy run playbook fix_containerd.yaml -y
fix_containerd.yaml:
---
- hosts: node control-plane
name: "Restore containerd config.toml"
tasks:
- name: check if /etc/containerd/config.toml.rpmsave exists
stat:
path: /etc/containerd/config.toml.rpmsave
register: containerd_config_toml_rpmsave
- name: check if /etc/containerd/config.toml exists
stat:
path: /etc/containerd/config.toml
register: containerd_config_toml
- name: Copy config.toml.rpmsave to config.toml iff there is no existing config.toml
become: yes
copy:
src: /etc/containerd/config.toml.rpmsave
dest: /etc/containerd/config.toml
remote_src: yes
owner: root
group: root
mode: '0644'
when:
- containerd_config_toml_rpmsave.stat.exists
- not containerd_config_toml.stat.exists