Hi @yhrn , sorry you ran into this. Our short term fix for you is to bump your CPU and memory limits. While I understand this is not an ideal solution and deployments should be meant to scale without issue, we are currently investigating performance improvements within Config Connector and will be gradually rolling out fixes. For this issue specifically, we are considering increasing/scaling the number of replicas defined in the ReplicaSet so that the workload of all your namespaces is distributed. I'll bring this up as an area for improvement!

Hi @caieo, thanks for the update. Regarding the suggested short term fix - will that work? Won't the operator just revert all objects it manages back to what it believes is the desired state?

Hi @yhrn, can you please try release 1.19.0, I am confident it will have a positive effect on the KCC components overall (but nothing specific to webhook-manager). Once we try that and the issue persists we can look deeper into webhook manager.

Hey @spew,

We're now on 1.19.1 and we are still having problems. The cnrm-webhook-manager keeps getting OOMKilled and we get intermittent failures like this trying to apply resources:

Internal error occurred: failed calling webhook "deny-unknown-fields.cnrm.cloud.google.com": Post https://cnrm-validating-webhook.cnrm-system.svc:443/deny-unknown-fields?timeout=29s

Hi @yhrn thanks so much for that log statement, that is helpful.

Hi @yhrn -- are you using Prometheus to get metrics?

We have "Cloud Operations for GKE" enabled so we get those metrics. Anything in specific you are looking for?

We've seen issues in the past with Prometheus putting load on the metrics-recorder process, just wanted to understand if there was some interaction there.

I've created a test cluster with 40 namespaces (and ~70 resources in each namespace) and I am not seeing any problems with the webhook-manager pod so there is still another variable I am missing to reproduce this.

Hey, we're happy to give you any additional data that might be helpful. One additional useful data point could be that it might be a memory leak of some kind (at least in 1.20): the process gets OOMKilled pretty regularly and the resources at death are pretty consistently: total-vm:789316kB, anon-rss:55400kB, file-rss:42520kB, shmem-rss:0kB. 770MiB is definitely greater than the pod limit which is 64MiB

We are making the following changes:

1. Changing the webhook's memory to a static, single value. For namespaced mode this value will be larger than before.
2. Webhook will generate its certificate once, on initial startup, and save it in a Secret in the cnrm-system namespace. When restarting or a new pod is created, the same value will be read out of the secret.
3. We will enable horizontal autoscaling on the webhook pod. The scaling threshold will be at 60% CPU usage.

The changes are available in version 1.21.1. Leaving this open for the time being until we can confirm it is enough to solve the issue.

We are on 1.23.0 (at least thats what annotations say in our GKE cluster with ConfigAddon enabled) and continue to see something similar to this problem: webhook pods are using 100% of their CPU quota and scaled to 10 instances with pod autoscaler.

Hi @redbaron, can you email me, rleidle@google.com, I would be curious as to the number of config connector resources you have and any information about the 'kinds' of resources would be good too.

You can see all your config-connector resources with kubectl get gcp --all-namespaces