feat(operator): add an opt-in pprof endpoint (PPROF_PORT)

chart/README.md

@@ -39,6 +39,7 @@ Settings | Description | Default |
 | controllerManager.manager.env.jobBackoffLimit | How many retries a package stage gets after its first attempt before its Job goes terminal, so a stage runs at most `jobBackoffLimit`+1 times. Only failures count: a pod lost to a disruption (eviction, preemption) spends nothing. "0" gives a single attempt with no retry. Interrupt stages other than a reboot get the same budget, counted in the interrupt container's restarts, which include restarts that were not failures, such as an unplanned node reboot: an interrupt that fails again with its restart count at `jobBackoffLimit` has its Job failed, and its node reads `erroring`, after about 30 seconds of kubelet restart backoff with the default "3". A reboot is not counted, because its own shutdown restarts it; `jobStageTimeout` bounds it instead; see [A failing interrupt](../docs/user-guide/custom-resource.md#a-failing-interrupt). Like `jobStageTimeout`, the value is fixed when a stage's Job is created. | "3" |
 | controllerManager.manager.env.legacyCleanupDelay | MIGRATION-SHIM (`skyhook.nvidia.com` → `nodewright.nvidia.com`): how long after a Skyhook finishes migrating the operator keeps its legacy node state, pods, and ConfigMap labels as a rollback window before pruning them. "0" prunes immediately. Removed at the removal release. | "24h" |
 | controllerManager.manager.env.publishLegacyMetrics | Keep exporting the deprecated `skyhook_*` metric series (with the `skyhook_name` label) alongside the current `nodewright_*` ones, so Grafana dashboards and Prometheus alerts written against the old names keep working across the rename. Set `"false"` to export only `nodewright_*`, which halves the operator's exported series count but breaks any consumer still querying the legacy names. The legacy series are removed unconditionally in operator v0.21.0. See [docs/observability/metrics.md](../docs/observability/metrics.md). | "true" |
+| controllerManager.manager.env.pprofPort | Serve Go pprof from the manager at this address when set; leave empty to keep it off. Bind to loopback only (for example `127.0.0.1:6060`) and reach it with `kubectl port-forward`: the endpoint is unauthenticated, and changing the value restarts the manager. See [docs/operations/resources-at-scale.md](../docs/operations/resources-at-scale.md). | "" |
 | controllerManager.manager.env.runtimeRequiredTaint | Nodes are expected to carry this taint, either by joining with it (e.g. the `--register-with-taints` kubelet flag) or via `autoTaintNewNodes: true` on the NodeWright, which makes the operator apply it. NodeWright **package** pods tolerate it automatically (the controller-manager pod does not; see `controllerManager.tolerations` and the note below), and the operator removes it from a node once every `runtimeRequired: true` NodeWright targeting that node has completed on it (completion on other nodes does not affect removal). During the rename deprecation window the operator additionally tolerates and removes the legacy `skyhook.nvidia.com=runtime-required:NoSchedule` taint, but never applies it. | nodewright.nvidia.com=runtime-required:NoSchedule | 
 | controllerManager.manager.image.repository | Where to get the image from | "ghcr.io/nvidia/nodewright/operator" |
 | controllerManager.manager.image.tag | what version of the operator to run | defaults to appVersion |

chart/templates/deployment.yaml

@@ -93,6 +93,10 @@ spec:
           value: {{ quote .Values.controllerManager.manager.env.metricsPort }}
         - name: PROBE_PORT
           value: {{ quote .Values.controllerManager.manager.env.probePort }}
+        {{- with .Values.controllerManager.manager.env.pprofPort }}
+        - name: PPROF_PORT
+          value: {{ quote . }}
+        {{- end }}
         - name: LEADER_ELECTION
           value: {{ quote .Values.controllerManager.manager.env.leaderElection }}
         - name: LOG_LEVEL

chart/values.yaml

@@ -50,6 +50,11 @@ controllerManager:
       logLevel: info
       metricsPort: :8443
       probePort: :8081
+      ## pprofPort serves Go's net/http/pprof from the manager when set. Leave empty to keep it
+      ## off. Bind to loopback only (for example 127.0.0.1:6060) and reach it with
+      ## kubectl port-forward: the endpoint is unauthenticated and the NetworkPolicy allows all
+      ## ingress. Changing it restarts the manager. See docs/operations/resources-at-scale.md.
+      pprofPort: ""
       ## reapplyOnReboot: "true" will reapply the packages on reboot. This is useful for systems that are read-only.
       reapplyOnReboot: "false"
       ## MIGRATION-SHIM (skyhook.nvidia.com -> nodewright.nvidia.com rename):

docs/operations/resources-at-scale.md

@@ -18,6 +18,31 @@ For the equations below the variables are as follows:
 | request | max(256Mi, N * 0.45) |
 | limit   | max(N *.8* max(P * 0.4, 1), 512) |
 
+## Profiling the operator
+
+When the operator's memory grows beyond what the equations above predict, `go_memstats_*` on the
+metrics port says how much is allocated but not which allocations. The manager can serve Go's
+`net/http/pprof` endpoints for that. It is off by default.
+
+Set `PPROF_PORT` (chart: `controllerManager.manager.env.pprofPort`) to a loopback address, for
+example `127.0.0.1:6060`, and reach it with a port-forward. The Deployment is named after the
+chart's fullname (`nodewright` by default; `nameOverride` and `fullnameOverride` change it), not
+the release name:
+
+```bash
+kubectl -n <namespace> port-forward deploy/nodewright-controller-manager 6060:6060
+go tool pprof -top -sample_index=inuse_space http://127.0.0.1:6060/debug/pprof/heap
+go tool pprof -top -sample_index=alloc_space http://127.0.0.1:6060/debug/pprof/heap
+curl -s 'http://127.0.0.1:6060/debug/pprof/profile?seconds=30' > cpu.pprof
+```
+
+Two cautions:
+
+* The endpoint is unauthenticated and the chart's NetworkPolicy allows all ingress, so bind to
+  `127.0.0.1` only. A value such as `:6060` exposes heap and CPU profiles to every pod in the cluster.
+* Changing the env var restarts the manager and discards the heap you wanted to inspect. Enable it
+  ahead of time on clusters where memory growth is expected, then profile once it has grown.
+
 ## cpu
 
 | item    | function |

operator/cmd/manager/main.go

@@ -102,6 +102,9 @@ type options struct {
 	LeaderElection bool `env:"LEADER_ELECTION, default=false"`
 	// EnableWebhooks Enables running of the webhook server, useful to disable for development
 	EnableWebhooks bool `env:"ENABLE_WEBHOOKS, default=true"`
+	// PprofPort The address the pprof endpoint binds to. Empty (the default) leaves it off.
+	// Set to 127.0.0.1:6060 to profile through kubectl port-forward.
+	PprofPort string `env:"PPROF_PORT, default="`
 
 	// zap logger settings, try to expose things from BindFlags into ENVs
 	LogEncoder      string `env:"LOG_ENCODER, default=json"`           // 'json' or 'console'
@@ -138,6 +141,7 @@ func main() {
 			FilterProvider: filters.WithAuthenticationAndAuthorization,
 		},
 		HealthProbeBindAddress: options.ProbePort,
+		PprofBindAddress:       options.PprofPort,
 		LeaderElection:         options.LeaderElection,
 		LeaderElectionID:       reconcileLeaseID,
 		// Scoping an informer to the operator namespace is only safe for kinds the operator

operator/config/manager/manager.yaml

@@ -104,6 +104,8 @@ spec:
             value: ":8443"
           - name: PROBE_PORT
             value: ":8081"
+          - name: PPROF_PORT
+            value: ""
           - name: LEADER_ELECTION
             value: "true"
           - name: LOG_LEVEL