Affected:
All versions of Konvoy
Issue Details
Alertmanager versions prior to v0.22.2 have known issues with firing some types of alerts. This version fixes some issues with sending alerts, such as Resolved alerts not being sent after an alert is no longer firing:
https://github.com/prometheus/alertmanager/commit/008b4a93da8138b851cea77ec068a97f3a440ba2
Alert merging assumed that EndsAt would always be empty for firing
alerts. This is no longer true starting with Prometheus v2.4.0: EndsAt
is set to a multiple of the evaluation interval or resend interval
(whichever is the largest).
If you are experiencing issues with some alerts not sending "resolved" notifications, you can bump the version of Alertmanager to version v0.22.2 or greater. We recommend you make this change via cluster.yaml and then run ./konvoy deploy addons. Here is an example of how you would bump the version of Alertmanager from cluster.yaml:
- name: prometheus enabled: true values: | alertmanager: alertmanagerSpec: image: tag: v0.22.2
You must then reconfigure alertmanager's notification settings as detailed here:
https://docs.d2iq.com/dkp/konvoy/1.7/monitoring/#notify-prometheus-alerts-in-slack