Overview/Background:
Environment: Applies to all provider
During Kommander install the gitea postgres DB is somehow missing the gitea role and the password auth is not working correctly. During the Gitea installation, the Postgres user and password are set via env vars on the Postgres deployment. And in some cases this process doesn't seem to have succeeded. You will see the gitea pod in Init:CrashLoopBackOff state
$ kubectl get po -A | grep gitea
kommander gitea-0 0/1 Init:CrashLoopBackOff 399 33h
And if you check the gitea-postgresql-0-gitea-postgresql.log you will see a repeating error below:
2022-03-10 14:42:10.121 GMT [89] LOG: database system was shut down at 2022-03-10 14:41:57 GMT
2022-03-10 14:42:10.135 GMT [1] LOG: database system is ready to accept connections
2022-03-10 14:42:18.201 GMT [102] FATAL: password authentication failed for user "gitea"
2022-03-10 14:42:18.201 GMT [102] DETAIL: Role "gitea" does not exist.
Connection matched pg_hba.conf line 1: "host all all 0.0.0.0/0 md5"
2022-03-10 14:42:28.205 GMT [108] FATAL: password authentication failed for user "gitea"
2022-03-10 14:42:28.205 GMT [108] DETAIL: Role "gitea" does not exist.
Connection matched pg_hba.conf line 1: "host all all 0.0.0.0/0 md5"
Resolution:
1. As an initial troubleshooting, we can simply try to restart the Postgres pod and see if this will help.
kubectl delete po -n kommander gitea-postgresql-0
2. If restarting the postgres pod didn't help, we can delete the Postgres PVC to wipe out its existing data.
NOTE: All Gitea data will be wiped out and any data in the existing PVC are not recoverable.
kubectl delete pvc data-gitea-postgresql-0 -n kommander
and try to get Postgres to re-initialize the database successfully:
kubectl delete po gitea-postgresql-0 -n kommander --now
3. Now we want the Gitea HelmRelease to try and reconcile again by running the command below. We can increase the number of {“retries”:xx} to a higher number and ultimately reconcile.
kubectl patch hr gitea -n kommander --patch=‘{“spec”:{“install”:{“remediation”:{“retries”:100}},“upgrade”:{“remediation”:{“retries”:100}}}}’ --type=merge
You can also check if the deployment status of the Gitea HelmRelease got successful.
kubectl get hr -A | grep gitea
40h
kommander gitea True Release reconciliation succeeded
4. Once we have the Gitea deployed successfully, we will then want to execute the command below to restart Kommander install.
kommander install <flags>
If the issue persist feel free to submit a case and upload fresh a diagnostic bundle.
https://docs.d2iq.com/dkp/konvoy/2.1/troubleshooting/generate-a-support-bundle/