Skip to content

fix(langfuse): raise clickhouse nofile limit and bound system log retention - #404

Open
kokhlo wants to merge 1 commit into
vxcontrol:mainfrom
kokhlo:fix/langfuse-clickhouse-nofile
Open

kokhlo wants to merge 1 commit into
vxcontrol:mainfrom
kokhlo:fix/langfuse-clickhouse-nofile

Conversation

@kokhlo

@kokhlo kokhlo commented Sep 8, 2026

Copy link
Copy Markdown

Summary

Fixes the "Too many open files" (errno 24) degradation of the bundled langfuse-clickhouse container described in #402: under sustained trace ingestion, background merges over the unbounded system.trace_log exhaust the Docker default nofile=4096, the failing merge is hot-retried forever at ~100% CPU per executor thread, and trace-detail queries return HTTP 500.

Changes

  1. docker-compose-langfuse.ymlulimits: nofile 262144/262144 on the clickhouse service. This is the root-cause fix (ClickHouse docs recommend ≥262k). Mirrors what docker-compose-observability.yml already sets for its clickhouse; works in rootless Docker. Adds a bind mount for a config.d override.
  2. langfuse/clickhouse/system-logs.xml (new) — bounded retention for system log tables: TTL 3 days for trace_log/text_log, 7 days for metric_log/query_log/part_log/processors_profile_log/asynchronous_metric_log; logger level warning to stop the debug flood. Without TTLs the giant merges keep being re-scheduled even after the FD limit is raised.

Validation

Closes #402

…ention

The clickhouse container ran with the Docker default nofile of 4096.
Background merges over the unbounded trace_log open thousands of sparse
subcolumn files at once, hit errno 24, and the failing merge is
hot-retried ~2.75x/s forever — pinning executor threads at 100% CPU and
breaking trace-detail queries with HTTP 500.

- ulimits nofile 262144/262144 on the clickhouse service, mirroring
  docker-compose-observability.yml (rootless-safe)
- system-logs.xml in config.d: TTL 3d for trace_log/text_log, 7d for
  metric/query/part/processors_profile/asynchronous logs, logger level
  warning to stop the debug flood
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant