Issue
Some users have reported that they get a “Bad Gateway” exception when they try to access the Konvoy, Kommander, and DKP Dashboards when using credentials stored in an external LDAP directory service.
The issue is experienced in scenarios where the list of groups the LDAP user belongs to is very large. It occurs because the complete list of groups for the user is fetched from LDAP and stored in a session cookie in the browser and there is a fixed limit for the size of such cookies.
To confirm whether this issue has been encountered, inspect the traefik-forward-auth (or traefik-forward-auth-mgmt) pod logs and confirm if the following error is reported:
time="2021-01-25T18:32:06Z" level=error msg="error saving session: securecookie: the value is too long" source_ip=10.7.68.151 2021/01/25 18:32:06 http: superfluous response.WriteHeader call from github.com/mesosphere/traefik-forward-auth/internal.(*Server).AuthCallbackHandler.func1 (server.go:319)
Solution
To solve this issue, the traefik-forward-auth component can be reconfigured to store the session cookies in etcD on the cluster instead of in the browser. How to do this depends on what version of DKP you are running:
Konvoy/Kommander 1.8.5:
The feature is enabled by adding the following setting to the traefik-forward-auth configuration in cluster.yaml:
- name: traefik-forward-auth
enabled: true
values: |
clusterStorage:
enabled: true
After making this change, each time a user logs in, a new secret is created in the kubeaddons namespace with a name that looks like this: tfa-claims-<...>.
Traefik Forward Auth includes a garbage collector that can clean up these secrets automatically after their lifetime is exhausted, which is set for 12 hours. However, the clusterRole that is provisioned for traefik-forward-auth is currently missing the correct permissions to delete these secrets. To enable the cleanup of secrets, you need to apply the following patch to the cluster after enabling clusterstorage:
kubectl patch clusterrole traefik-forward-auth-kubeaddons --type='json' -p='[{"op": "add", "path": "/rules/0", "value": {"apiGroups": [""], "resources":["secrets"], "verbs": ["get","list","update","create","delete"]}}]'
DKP 2.1 and higher:
The clusterstorage feature is enabled by adding the following setting to the traefik-forward-auth-mgmt configuration when installing Kommander:
traefik-forward-auth-mgmt:
values: |
clusterStorage:
enabled: true
namespace: kommander
After making this change, each time a user logs in, a new secret is created in the kubeaddons namespace with a name that looks like this: tfa-claims-<...>..
Traefik Forward Auth includes a garbage collector that can clean up these secrets automatically after their lifetime is exhausted, which is set for 12 hours. However, in DKP 2.1, the clusterRole that is provisioned for traefik-forward-auth is currently missing the correct permissions to delete secrets. DKP 2.2 added the necessary permission.
If you are still using DKP 2.1, to enable the cleanup of secrets, you need to apply the following patch to the cluster after enabling clusterStorage:
kubectl patch clusterrole traefik-forward-auth-mgmt --type='json' -p='[{"op": "add", "path": "/rules/0", "value": {"apiGroups": [""], "resources":["secrets"], "verbs": ["get","list","update","create","delete"]}}]'