Skip to main content

Custom Domains, ACME certificates and Custom CA certificates

Anthony Windebank
  • Updated

Most people think of solutions like LetsEncrypt when implementing an ACME solution for their DKP clusters, and we provide documentation on how to set this up here:

https://docs.d2iq.com/dkp/latest/configuration-example-with-let-s-encrypt


However, you might be wondering how you can leverage ACME certificates in an environment where external solutions like LetsEncrypt are not possible due to network restrictions like Air Gap. You might not have internet access, but your environment might already be set up with a private ACME certificate server such as SmallStep CA server. Since you don't have internet access, you probably also are using a local Certificate Authority to create non ACME certificates to secure your local hosts and services, such as your local ACME server, LDAP server etc. But how do you configure DKP to take advantage of your local security infrastructure?

 

For this example we will assume that you have already set up a local Certificate Authority such as the one that is bundled with pFsense, and that you have created a CA certificate via this service. We will call this certificate AirGapCA.crt. 

We will also assume that you have set up a local ACME server, using the above CA certificate so that your ACME server can sign certificates on behalf of your Certificate Authority. 

 

Step 1 - Deploy DKP 2.4.0 or Higher:

 

Ensure that AirGapCA.crt is pushed to your local trust store on all hosts you plan to deploy DKP to. You can do this manually or use an ansible playbook similar to the below:

 - name: configure hosts for AirGap
   hosts: all
   become: yes
   tasks:
   - name: push CA certificate to hosts
     copy:
       src: certs/AirGapCA.crt
       dest: /etc/pki/ca-trust/source/anchors/AirGapCA.crt
   - name: update the CA trust
     command: 
       cmd: update-ca-trust


Once your hosts trust your local CA certificate, you can deploy DKP as normal. Do not deploy Kommander yet.

 

Step 2 - Set up your ClusterIssuer:

We want to use a Custom Domain for our DKP cluster and we want DKP to automatically request an ACME certificate for this custom domain to ensure our local environment is secure. We will craft a special ClusterIssuer object that will be leveraged by the Kommander installation process to handle all of this for us. 

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: kommander-acme-issuer
spec:
  acme:
    email: tony@coffeeshop.local
    server: https://acme.coffeshop.local/acme/acme/directory
    privateKeySecretRef:
      name: coffeeshop-issuer-private-key
    solvers:
    - http01:
        ingress:
          ingressTemplate:
            metadata:
              annotations:
                 kubernetes.io/ingress.class: kommander-traefik
                 traefik.ingress.kubernetes.io/router.tls: "true"

 

We are specifically creating this ClusterIssuer before Kommander installation and calling the ClusterIssuer kommander-acme-issuer because during the Kommander installation, if this ClusterIssuer object already exists, then Kommander will use the configuration we specify here instead of the default config.

 

We must also tell Kommander to use our local CA certificate AirGapCA.crt when it is requesting the ACME certificate so that cluster applications such as dex-k8s-authenticator, traefik-forward-auth and kube-oidc-proxy trust the legitimacy of the ACME certificate. We can create a generic secret using AirGapCA.crt with a specific name - kommander-traefik-tls - and Kommander will re-use this secret to dump the tls.crt and tls.key from our ACME request into it. 

 

kubectl create namespace kommander
kubectl create secret generic -n kommander kommander-traefik-tls --from-file=ca.crt=AirGapCA.crt

 

Step 3 - Generate and Configure Kommander.yaml


We're almost ready to deploy Kommander, but first we have to tell Kommander about our ACME server and custom domain. 

Generate a kommander.yaml file:

./dkp install kommander --init > kommander.yaml

 

Edit the file to change the custom domain and add entries for acme:

apiVersion: config.kommander.mesosphere.io/v1alpha1
kind: Installation
clusterHostname: "securecluster.coffeeshop.local"
acme:
  email: tony@coffeeshop.local
  server: https://acme.coffeshop.local/acme/acme/directory


Note that by default there is a blank clusterHostname: "" value at the bottom of kommander.yaml, ensure that you are not defining the same entry twice in the same file. If you are using DKP 2.4.0 or above on prem, this is a good time to configure the rook-ceph application values: 

https://support.d2iq.com/hc/en-us/articles/12618140300948-Rook-Ceph-Installation-Errors-for-On-Premise

 

Step 4 - Deploy Kommander


Run the below command to deploy Kommander with our custom ACME settings:

 

./dkp install kommander --installer-config kommander.yaml

 

Kommander will automatically request a certificate from the preconfigured ClusterIssuer. The ClusterIssuer will create a challenge object that you can view via:

 

kubectl get challenge -A


As part of this challenge, a Solver Pod will be created in the Kommander namespace as well as a service and ingress object to help route to it. Once the challenge is complete all related objects will disappear from the cluster quickly. You can check if the certificate is ready via the following command:

 

kubectl get certificate -n kommander kommander-traefik-tls


Your certificate should show as Ready: True

[tony@rockyadmin cluster-a]$ kubectl get certificate -n kommander kommander-traefik-tls
NAME READY SECRET AGE
kommander-traefik-tls True kommander-traefik-tls 7h57m


We also need to ensure that the following helm releases were able to reconcile:

dex-k8s-authenticator

traefik-forward-auth-mgmt

kube-oidc-proxy


If the secret kommander-traefik-tls was not configured properly, then dex-k8s-authenticator pods may have issues starting. You can check the logs of the pod to verify this is the issue:

dex-k8s-authenticator pod errors:

[tony@rockyadmin cluster-a]$ kubectl logs -n kommander dex-k8s-authenticator-589bc8f965-bxgs9
2023/02/02 22:00:37 Using config file: /app/configuration/config.yaml
2023/02/02 22:00:37 useClusterHostname: true
2023/02/02 22:00:37 Creating new provider https://securecluster.coffeeshop.local/dex
2023/02/02 22:00:37 Failed to query provider "https://securecluster.coffeeshop.local/dex": Get https://securecluster.coffeeshop.local/dex/.well-known/openid-configuration: x509: certificate signed by unknown authority

 

traefik-forward-auth-mgmt pod errors:

[tony@rockyadmin cluster-a]$ kubectl logs -n kommander traefik-forward-auth-mgmt-795c494bc7-q7tx9 8time="2023-02-22T07:43:07Z" level=fatal msg="failed to get provider configuration for https://skeletonjelly.daclusta/dex: Get \"https://securecluster.coffeeshop.local/dex/.well-known/openid-configuration\": x509: certificate signed by unknown authority (hint: make sure https://securecluster.coffeeshop.local/dex is accessible from the cluster)"

 

If you do encounter an error, you can always modify the kommander-traefik-tls secret to update the ca.crt value and then perform a rollout restart of kommander deployments to resolve the issue. You should now have a Custom Domain successfully configured with an automatically generated and renewed ACME certificate as part of the Kommander deployment process!