You may notice that your Konvoy cluster is frequently triggering one of the default Prometheus alert rules, "etcdHighNumberOfFailedGRPCRequests".
In most cases, this is a false positive caused by a metric that is incorrectly reported by etcd. If you do not notice other alerts going off, or other negative symptoms in your cluster, you may also be affected by this bug.
See here for more details on the issue: https://github.com/etcd-io/etcd/issues/10289
Versions of Kubernetes that include etcd v3.4.16 and later should have this bug fixed, so in many cases you can resolve the root cause of this issue by upgrading.
However, if you'd like to remove the rule from your Konvoy cluster in the meantime, you can run:
kubectl -n kubeaddons edit prometheusrules prometheus-kubeaddons-prom-etcd
and remove the YAML block belonging to the etcdHighNumberOfFailedGRPCRequests alert.