Skip to content

feat: add on-demand pprof server and heap auto-dump via environment variables - #475

Open
david-garcia-garcia wants to merge 2 commits into
crowdsecurity:mainfrom
david-garcia-garcia:profileer
Open

feat: add on-demand pprof server and heap auto-dump via environment variables#475
david-garcia-garcia wants to merge 2 commits into
crowdsecurity:mainfrom
david-garcia-garcia:profileer

Conversation

@david-garcia-garcia

@david-garcia-garcia david-garcia-garcia commented May 11, 2026

Copy link
Copy Markdown

What

Adds pkg/profiling — a zero-dependency package (stdlib only) that provides two opt-in profiling features, both controlled exclusively by environment variables so they can be turned on in a running container without rebuilding the image or editing config files.

Fixes #476

pprof HTTP server (CS_PROFILING_ENABLED=true)

  • Starts /debug/pprof/* on a dedicated listener (default :6060)
  • Uses its own http.ServeMux — pprof handlers are never registered on http.DefaultServeMux and cannot leak onto the Prometheus /metrics server
  • Logs listen address + a hint to set GODEBUG=gctrace=1 on start

Heap auto-dump watcher (CS_PROFILING_HEAP_DUMP_DIR=<path>)

  • Background goroutine polls runtime.ReadMemStats every CS_PROFILING_HEAP_POLL_INTERVAL (default 30s)
  • Writes a gzipped pprof heap profile (heap-<RFC3339>.pb.gz) when HeapAlloc exceeds CS_PROFILING_HEAP_DUMP_THRESHOLD_MB (default 200 MiB)
  • Respects a cooldown (CS_PROFILING_HEAP_DUMP_COOLDOWN, default 5m) to avoid flooding the volume
  • Exits cleanly on context cancellation

Both features are disabled by default — existing deployments are unaffected.

Why

The process intermittently spikes from ~30 MB to >300 MB and gets OOM-killed in container deployments. There was no way to collect a heap profile during or after a spike.

Changes

  • pkg/profiling/profiling.go — new package
  • cmd/root.go — two call sites after the Prometheus block
  • `README

@blotus

blotus commented Jun 2, 2026

Copy link
Copy Markdown
Member

Hello,

Thank you for the PR.

A few comments:

  • To be more consistent with how pprof is exposed in crowdsec, the server should be exposed on the same port as prometheus (if prometheus is disabled, then no pprof server)
  • I'm not sure that the auto-dump is needed: it's something that seems like it could be easily replicated out of the process (query the prometheus metrics -> get the current memory usage -> query the pprof heap dump endpoint if needed)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add on-demand pprof profiling and heap auto-dump, controlled via environment variables

2 participants