Kommander can attach another kubernetes cluster to manage it. It can also federate addons to the target cluster for management and observation. How does the cluster attachment process work, and how does the federation of addons function? How do we debug issues during the attachment and federation process?
After attaching a cluster via the CLI or the Kommander Web UI, you should see your cluster attach with a status of "Active". You can also find this information from kubectl:
We can see that the above cluster "alpha" has been attached and has a Ready status of "True".
We can also describe this kubefedcluster object to get more information about how it is being managed:
Looking at the labels for this object, we can see exactly what addons are being federated, as well as the type of cluster:
We can also see more information about the health of the cluster under status:
One very important label for kubefedcluster objects is the kommander.mesosphere.io/cluster-type label. This label is set by Kommander when attaching a cluster. Kommander looks at the labels on the target cluster's nodes to determine what type of cluster it is. A cluster will be labeled with
kommander.mesosphere.io/cluster-type=Konvoy
If the node label konvoy.mesosphere.com/inventory_hostname is present. This is important because Kommander will not federate addons such as cert-manager to a cluster that it detects as type Konvoy. You cannot modify this label once a cluster has been attached.
If the target is a konvoy cluster, you can use the following command to check out which addons are being deployed:
If the target is a non-konvoy cluster. you need to more specific to get the right information:
What if you don't see an addon in the above list, but you expect it to be there? Kommander uses selectors to determine what addons get federated out. We need to look at these selectors to identify the reason a specific addon might not be pushed to our target cluster. We can get a full list of the addons that can be federated with the following command:
If we examine a specific addon, we can more information on the selectors:
Namespaced addon:
Global clusteraddon:
Looking under Spec.Placement we can see what selectors are used:
We can see that for the above example, only Kubernetes versions 1.16-1.20 are supported, so if the target cluster is outside of this range, Kommander won't federate the addon!
After attaching a cluster via the CLI or the Kommander Web UI, you should see your cluster attach with a status of "Active". You can also find this information from kubectl:
kubectl get kubefedcluster -A NAMESPACE NAME AGE READY kommander alpha 7m58s True
We can see that the above cluster "alpha" has been attached and has a Ready status of "True".
We can also describe this kubefedcluster object to get more information about how it is being managed:
kubectl describe kubefedcluster -n kommander alpha
Looking at the labels for this object, we can see exactly what addons are being federated, as well as the type of cluster:
Name: alpha
Namespace: kommander
Labels: kommander.mesosphere.io/cluster-id=373db218-9160-4cd1-bf7a-0d40580671d8
kommander.mesosphere.io/cluster-type=Konvoy
kommander.mesosphere.io/federate-cert-manager=default-true
kommander.mesosphere.io/federate-elasticsearch=default-false
kommander.mesosphere.io/federate-elasticsearch-curator=default-false
kommander.mesosphere.io/federate-elasticsearchexporter=default-false
kommander.mesosphere.io/federate-fluentbit=default-false
kommander.mesosphere.io/federate-ingress-overrides=default-true
kommander.mesosphere.io/federate-kibana=default-false
kommander.mesosphere.io/federate-kube-oidc-proxy=default-true
kommander.mesosphere.io/federate-kubeaddons=default-true
kommander.mesosphere.io/federate-kubecost=default-true
kommander.mesosphere.io/federate-prometheus=default-true
kommander.mesosphere.io/federate-prometheusadapter=default-true
kommander.mesosphere.io/federate-reloader=default-true
kommander.mesosphere.io/federate-traefik=default-true
kommander.mesosphere.io/federate-traefik-forward-auth=default-true
kommander.mesosphere.io/federation-enabled=true
kommander.mesosphere.io/kubeaddons-composite-values-v0.16=true
kommander.mesosphere.io/kubeaddons-version=v0.26.1
kommander.mesosphere.io/kubernetes-version=v1.20
workspaces.kommander.mesosphere.io/workspace-ref=7432e8ec-d0b8-4503-8a0b-46f61d20fc4c
We can also see more information about the health of the cluster under status:
Status:
Conditions:
Last Probe Time: 2021-09-30T21:54:07Z
Last Transition Time: 2021-09-30T21:46:06Z
Message: /healthz responded with ok
Reason: ClusterReady
Status: True
Type: Ready
Events:
One very important label for kubefedcluster objects is the kommander.mesosphere.io/cluster-type label. This label is set by Kommander when attaching a cluster. Kommander looks at the labels on the target cluster's nodes to determine what type of cluster it is. A cluster will be labeled with
kommander.mesosphere.io/cluster-type=Konvoy
If the node label konvoy.mesosphere.com/inventory_hostname is present. This is important because Kommander will not federate addons such as cert-manager to a cluster that it detects as type Konvoy. You cannot modify this label once a cluster has been attached.
If the target is a konvoy cluster, you can use the following command to check out which addons are being deployed:
kubectl get addons,clusteraddons -A
If the target is a non-konvoy cluster. you need to more specific to get the right information:
kubectl get addon.kubeaddons.mesosphere.io,clusteraddon.kubeaddons.mesosphere.io -A
NAMESPACE NAME READY STAGE REVISION
kommander-system addon.kubeaddons.mesosphere.io/kommander-kubeaddons-kommander-karma-proxy false deploying
kommander-system addon.kubeaddons.mesosphere.io/kommander-kubeaddons-kommander-thano-proxy false deploying
kommander-system addon.kubeaddons.mesosphere.io/kommander-kubeaddons-kubecost-proxy false deploying
kommander-system addon.kubeaddons.mesosphere.io/kube-oidc-proxy-kommander
kommander-system addon.kubeaddons.mesosphere.io/traefik-forward-auth-kommander
kubeaddons addon.kubeaddons.mesosphere.io/prometheus
kubeaddons addon.kubeaddons.mesosphere.io/prometheusadapter false
kubeaddons addon.kubeaddons.mesosphere.io/reloader true deployed
NAMESPACE NAME READY STAGE REVISION
clusteraddon.kubeaddons.mesosphere.io/cert-manager false deploying
clusteraddon.kubeaddons.mesosphere.io/kubecost
clusteraddon.kubeaddons.mesosphere.io/traefik false
What if you don't see an addon in the above list, but you expect it to be there? Kommander uses selectors to determine what addons get federated out. We need to look at these selectors to identify the reason a specific addon might not be pushed to our target cluster. We can get a full list of the addons that can be federated with the following command:
kubectl get federatedaddons,federatedclusteraddons -A
NAMESPACE NAME AGE
kommander-system federatedaddon.types.kubefed.io/kommander-kubeaddons-kommander-karma-proxy 119m
kommander-system federatedaddon.types.kubefed.io/kommander-kubeaddons-kommander-thano-proxy 119m
kommander-system federatedaddon.types.kubefed.io/kommander-kubeaddons-kubecost-proxy 119m
kommander-system federatedaddon.types.kubefed.io/kube-oidc-proxy-kommander 119m
kommander-system federatedaddon.types.kubefed.io/traefik-forward-auth-kommander 119m
kubeaddons federatedaddon.types.kubefed.io/elasticsearch 119m
kubeaddons federatedaddon.types.kubefed.io/elasticsearch-curator 119m
kubeaddons federatedaddon.types.kubefed.io/elasticsearchexporter 119m
kubeaddons federatedaddon.types.kubefed.io/fluentbit 119m
kubeaddons federatedaddon.types.kubefed.io/kibana 119m
kubeaddons federatedaddon.types.kubefed.io/prometheus 119m
kubeaddons federatedaddon.types.kubefed.io/prometheusadapter 119m
kubeaddons federatedaddon.types.kubefed.io/reloader 119m
NAMESPACE NAME AGE
federatedclusteraddon.types.kubefed.io/cert-manager 119m
federatedclusteraddon.types.kubefed.io/kubecost 119m
federatedclusteraddon.types.kubefed.io/traefik 119m
If we examine a specific addon, we can more information on the selectors:
Namespaced addon:
kubectl describe federatedaddon reloader -n kubeaddons
Global clusteraddon:
kubectl describe federatedclusteraddon cert-manager
Looking under Spec.Placement we can see what selectors are used:
Spec:
Placement:
Cluster Selector:
Match Expressions:
Key: kommander.mesosphere.io/cluster-type
Operator: Exists
Values:
Key: kommander.mesosphere.io/cluster-type
Operator: NotIn
Values:
Konvoy
Key: kommander.mesosphere.io/federate-cert-manager
Operator: In
Values:
true
default-true
Key: kommander.mesosphere.io/kubernetes-version
Operator: In
Values:
v1.16
v1.17
v1.18
v1.19
v1.20
We can see that for the above example, only Kubernetes versions 1.16-1.20 are supported, so if the target cluster is outside of this range, Kommander won't federate the addon!