Kubernetes 1.37 makes webhook concurrency an upgrade test

The upgrade changes the shape of the load

A conversion webhook that works reliably when calls arrive one at a time may behave differently when the API server calls it concurrently. That is the upgrade risk I would test before moving to Kubernetes 1.37.

The Kubernetes 1.37 release announcement describes concurrent decoding and transformation of watch events during watch-cache initialization. For CRDs that need conversion, this can mean up to ten conversion calls running concurrently against a webhook during cache initialization, rather than one at a time. The announcement says the total call volume is unchanged; the change is how many calls can be in flight together.

That distinction matters. A webhook can process the same work successfully in total and still fail when the work is concentrated into a shorter interval. A limited worker pool, a connection limit, a serialized code path, or a downstream database that cannot absorb parallel queries can turn concurrent requests into longer queues, timeouts, or errors. Those are risks to check in your own system, not outcomes guaranteed by the release.

I would make this a workload-level upgrade check, not just a question of whether the control plane starts. The release makes cache initialization a specific time to exercise the conversion path under parallel load. If a CRD is served in a version different from its stored version, conversion may be needed when the API server reads those objects; the Kubernetes documentation on CRD version conversion explains how webhook conversion fits into that process.

Test the path you actually depend on

Start by identifying which CRDs use webhook conversion and which version transitions matter in your cluster. A webhook may be healthy in a basic readiness check but still fail on a conversion request that exercises a particular resource shape or version pair. The test should use representative objects and the conversion directions your clients and controllers actually rely on.

Then run the check in an environment where you can observe API-server cache initialization without putting production availability at risk. Exercise the webhook while the relevant cache is cold or initializing, and make calls overlap rather than arriving serially. The point is not to reproduce a whole cluster perfectly; it is to see whether the conversion service and its dependencies behave correctly when concurrent work reaches them.

I would record at least these signals during the test:

What to observe	What it can reveal
In-flight requests and queue depth	Whether parallel calls exceed the webhook’s worker or connection capacity
Request latency, including tail latency	Whether queuing or downstream contention makes conversions too slow
Conversion errors and timeouts	Whether the service fails under overlap, even if individual requests work alone
CPU, memory, and downstream resource use	Whether the webhook’s dependencies become the actual bottleneck
API-server errors and cache initialization progress	Whether webhook trouble affects the control-plane operation being tested

These are observations, not pass criteria by themselves. Set thresholds from the service’s existing SLOs and operational limits. If the webhook has a deliberate concurrency cap, test that cap under the expected burst instead of assuming the Kubernetes-side change will respect it. If the webhook fans out to a database or another service, include those dependencies in the observation: a fast handler that transfers the queue downstream has not solved the capacity problem.

Decide what failure looks like before the test

A successful test is not just one where the webhook returns valid conversions. I would also decide in advance what behavior counts as a failure: latency beyond the webhook’s budget, growing queues that do not drain, resource saturation, conversion errors, or API-server initialization that stalls. A test without explicit failure signals can pass while merely shifting the bottleneck somewhere less visible.

I would test failure behavior deliberately in a controlled environment. For example, observe what happens when the webhook approaches its capacity limit or a dependency becomes slow. Does the service reject or time out requests predictably? Does recovery clear the queue, or do retries and overlapping work keep it saturated? The goal is not to provoke an outage for its own sake. It is to find out whether the system fails in a bounded, diagnosable way before the upgrade makes this load pattern relevant.

Be careful not to apply admission-webhook settings to conversion webhooks by analogy. Kubernetes documents failurePolicy and timeoutSeconds for admission webhooks, but those are not a general configuration switch for CRD conversion. For conversion, inspect the CRD’s webhook configuration and the behavior of the conversion service itself; the CRD API reference documents the conversion webhook fields. The useful question is how a failed or slow conversion affects the API operations that need it, not whether a setting with a familiar name exists elsewhere.

Make the result an upgrade gate

I would treat the result as a concrete upgrade gate for the webhook, separate from general release readiness. The release notes provide a workload-specific change to test: concurrent conversion calls during cache initialization. That makes it possible to ask a direct question of the system: can this webhook, with its actual limits and dependencies, handle overlapping conversion work without unacceptable latency or failure?

If it cannot, I would fix the bottleneck or adjust capacity before upgrading. If it can, keep the test and its observations with the upgrade evidence so the result is repeatable when the webhook, its dependencies, or the cluster changes. The relevant result is not that Kubernetes 1.37 initializes its cache faster; it is that the conversion service stays within its own operating limits when initialization sends work in parallel.