Problem

Every environment is different, and in some cases, it may be a requirement to modify DKP applications to be better tailored to your unique environment. In the example below, a manual change has been made to Ingestor statefulset so that instead of a 10Gi claim, it is requesting a 20Gi claim:

volumeClaimTemplates: 
- apiVersion: v1 
  kind: PersistentVolumeClaim 
  metadata: 
    creationTimestamp: null 
    name: data 
  spec: 
    accessModes: 
    - ReadWriteOnce 
    resources: 
      requests: 
        storage: 20Gi

While upgrading DKP, Flux will check for any override configmaps for specific configurations, and if none are found, then it will apply all of the values from the D2IQ default configmap. Because of this, if you have manually altered an immutable field, you will be met with the below while upgrading:

error updating the resource "grafana-loki-loki-distributed-ingester":
cannot patch "grafana-loki-loki-distributed-ingester" with kind StatefulSet: 
StatefulSet.apps "grafana-loki-loki-distributed-ingester" is invalid: 
spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden

Solution

In this situation, you have a few different ways to upgrade. If you unknowingly applied the manual change and did not wish to have it persist, you can delete the object with kubectl delete statefulset grafana-loki-loki-distributed-ingester --cascade=orphan. This will delete the statefulset object but allow the current pods to remain while the new statefulset is created and the pods from that object are spun up. In this case, it will revert back to the default value of 10Gi for its volumeClaim. If instead, you wish to keep your custom configuration, we must then add the values you have configured to your helmrelease override configmap. You can validate the name of the override configmap by running kubectl describe hr -n kommander grafana-loki. In the output, you will see a 'valuesFrom' section:

Release Name: grafana-loki
Upgrade:
  Crds: CreateReplace
  Remediation:
    Retries: 30
Values From:
  Kind: ConfigMap
  Name: grafana-loki-0.48.5-d2iq-defaults
  Kind: ConfigMap
  Name: loki-overrides

If you do not see an overrides configmap, you can create one by manually adding the override to the appdeployment object by following the below steps:

Create your configmap:

cat loki-cm.yaml
apiVersion: v1
data:
  values.yaml: |
    ingester:
      persistence:
      size: 20Gi
kind: ConfigMap
metadata:
  name: loki-overrides
  namespace: kommander

kubectl apply -f loki-cm.yaml

Gather the appdeployment YAML:

kubectl get appdeployment -n grafana-loki -o yaml -n kommander > loki.yaml

Edit the YAML to contain the override:

apiVersion: apps.kommander.d2iq.io/v1alpha3
kind: AppDeployment
metadata:
  finalizers:
  - kommander.mesosphere.io/appdeployment
  - kommander.mesosphere.io/appsconfigfederation
  name: grafana-loki
  namespace: kommander
spec:
  appRef:
    kind: ClusterApp
    name: grafana-loki-0.48.5
  configOverrides:
    name: loki-overrides

Apply override YAML:

kubectl apply -f loki.yaml

After some time, Flux will automatically reconcile the helmrelease object; when this happens, the override configuration will now be in line with our manually applied changes, witch will allow the upgrade to get pas the failure to patch error.