Skip to main content

Why do I get a "Bad Gateway" exception when trying to access the DKP dashboard via a Dex LDAP connector?

Jena Miller
  • Updated

Issue

Some users have reported that they get a "Bad Gateway" exception when they try to access the DKP Dashboards when using credentials stored in an external LDAP directory service. 

The issue is experienced in scenarios where the list of groups the LDAP user belongs to is very large.  It occurs because the complete list of groups for the user is fetched from LDAP and stored in a session cookie in the browser and there is a fixed limit for the size of such cookies.

To confirm whether this issue has been encountered, inspect the traefik-forward-auth (or traefik-forward-auth-mgmt) pod logs and confirm if the following error is reported:

time="2021-01-25T18:32:06Z" level=error msg="error saving session: securecookie: the value is too long" source_ip=10.7.68.151
2021/01/25 18:32:06 http: superfluous response.WriteHeader call from github.com/mesosphere/traefik-forward-auth/internal.(*Server).AuthCallbackHandler.func1 (server.go:319)

Solution

To solve this issue, the traefik-forward-auth component can be reconfigured to store the session cookies in etcd on the cluster instead of in the browser. 

The clusterstorage feature is enabled by adding the following setting to the traefik-forward-auth-mgmt configuration when installing Kommander:

 traefik-forward-auth-mgmt:
   values: |
     clusterStorage:
       enabled: true
       namespace: kommander

After making this change, each time a user logs in, a new secret is created in the kommander namespace with a name that looks like this:  tfa-claims-<...>..   
 
Traefik Forward Auth includes a garbage collector that can clean up these secrets automatically after their lifetime is exhausted, which is set for 12 hours. However, in DKP 2.1, the clusterRole that is provisioned for traefik-forward-auth is currently missing the correct permissions to delete secrets.  DKP 2.2 added the necessary permission.

If you are still using DKP 2.1, to enable the cleanup of secrets, you need to apply the following patch to the cluster after enabling clusterStorage:

kubectl patch clusterrole traefik-forward-auth-mgmt --type='json' -p='[{"op": "add", "path": "/rules/0", "value": {"apiGroups": [""], "resources":["secrets"], "verbs": ["get","list","update","create","delete"]}}]'