Skip to main content

Resolving issues with HelmReleases that are failed.

Craig Neth
  • Updated

Overview

There is an existing issue with the Flux helm-controller (https://github.com/fluxcd/helm-controller/issues/149) that can cause HelmReleases to get "stuck" with an error message like "Helm upgrade failed: another operation (install/upgrade/rollback) is in progress". This can happen anytime the helm-controller is restarted while a HelmRelease is upgrading/installing/etc.

Workaround

To ensure the HelmRelease error was caused by the helm-controller restarting, first try to suspend/resume the HelmRelease:

kubectl -n <namespace> patch helmrelease <HELMRELEASE_NAME> --type='json' -p='[{"op": "replace", "path": "/spec/suspend", "value": true}]'
kubectl -n <namespace> patch helmrelease <HELMRELEASE_NAME> --type='json' -p='[{"op": "replace", "path": "/spec/suspend", "value": false}]'

It's possible this will resolve the issue.   

You should see the HelmRelease attempting to reconcile, and then it will either succeed (with status: 'Release reconciliation succeeded') or it will fail with the same error as before, i.e: "Helm upgrade failed: another operation (install/upgrade/rollback) is in progress". 

If the HelmRelease is still in the failed state, it's likely related to the helm-controller restarting. To resolve the issue, do the following.   For example, if the 'reloader' HelmRelease is the one that is stuck:

1. List secrets containing the affected HelmRelease name:

kubectl get secrets -n ${NAMESPACE} | grep reloader

kommander-reloader-reloader-token-9qd8b                        kubernetes.io/service-account-token   3      171m
sh.helm.release.v1.kommander-reloader.v1                       helm.sh/release.v1                    1      171m
sh.helm.release.v1.kommander-reloader.v2                       helm.sh/release.v1                    1      117m

In the example above, "sh.helm.release.v1.kommander-reloader.v2" is the most recent revision.

2. Find and delete the most recent revision secret (i.e. sh.helm.release.v1.*.<revision>)

kubectl delete secret -n <namespace> <most recent helm revision secret name>

3. suspend/resume the HelmRelease to trigger a reconciliation

kubectl -n <namespace> patch helmrelease <HELMRELEASE_NAME> --type='json' -p='[{"op": "replace", "path": "/spec/suspend", "value": true}]'
kubectl -n <namespace> patch helmrelease <HELMRELEASE_NAME> --type='json' -p='[{"op": "replace", "path": "/spec/suspend", "value": false}]'

4. You should see the HelmRelease get reconciled then eventually the upgrade/install will succeed.