perf(testcontainers): replace rabbitmq-diagnostics healthcheck with raw TCP probe

[fatih-acar]

Summary

The aggressive 1s RabbitMQ healthcheck cadence in infrahub-testcontainers silently costs more than two CPU cores per broker container for the entire life of every test stack: each rabbitmq-diagnostics invocation boots a full Erlang VM (~2.1s of CPU time measured on rabbitmq:4.2.1-management). On CI machines running several stacks in parallel, that CPU is stolen from the tests themselves. This PR keeps the aggressive interval but makes each check effectively free.

Key Changes

• The message-queue healthcheck in both testcontainers compose files (single-node and cluster) now uses a raw TCP probe (bash -c '</dev/tcp/127.0.0.1/5672') instead of rabbitmq-diagnostics -q check_port_connectivity, cutting per-check cost from ~2.1s CPU to ~1ms (~2000x).
• The readiness signal is identical: check_port_connectivity only verifies that listener ports accept TCP connections, and RabbitMQ opens the AMQP listener at the very end of boot.
• Removes a latent flake: the Erlang CLI's 0.3–1s+ wall time raced against the 1s healthcheck timeout under CPU contention; the TCP probe completes instantly.
• Exec form (["CMD", "bash", ...]) is required because the image's /bin/sh is dash, which lacks /dev/tcp.

Validation

Verified empirically against rabbitmq:4.2.1-management:

• rabbitmq-diagnostics -q check_port_connectivity: 2.15s CPU (user+sys) per run; bash TCP probe: ~1ms.
• Repeated bare TCP probes on 5672 produce zero broker log lines (RabbitMQ does not log connections closed before the protocol header).
• A stack booted with the exact new healthcheck block reaches healthy via docker compose up --wait in ~2.6s (first probe fails mid-boot, second succeeds).

Test Plan

• E2E and integration CI suites boot their stacks through these compose files via docker compose up --wait; a green run exercises the new healthcheck end-to-end.

🤖 Generated with Claude Code

[cubic-dev-ai]

1 issue found across 2 files

Confidence score: 4/5

• In python_testcontainers/infrahub_testcontainers/docker-compose-cluster.test.yml, the broker healthcheck assumes bash exists even though MESSAGE_QUEUE_DOCKER_IMAGE is configurable; Alpine or custom images may fail health checks and prevent the stack from starting. Make the healthcheck shell-compatible with supported images or constrain/document the image requirement.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="python_testcontainers/infrahub_testcontainers/docker-compose-cluster.test.yml">

<violation number="1" location="python_testcontainers/infrahub_testcontainers/docker-compose-cluster.test.yml:40">
P2: This healthcheck hard-requires bash in the broker image, but the image is configurable via MESSAGE_QUEUE_DOCKER_IMAGE. If anyone runs the stack against a non-Debian rabbitmq image (e.g. an alpine variant or a custom build) that lacks bash, the exec-form probe fails to find bash, the healthcheck reports unhealthy, and depends_on: service_healthy for message-queue blocks the whole stack from starting. The previous rabbitmq-diagnostics form worked on any rabbitmq image. Note the fragility in the comment or fall back to a form that doesn't depend on bash when /bin/sh is dash.</violation>
</file>

<sub>Shadow auto-approve: would not auto-approve because issues were found.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: This healthcheck hard-requires bash in the broker image, but the image is configurable via MESSAGE_QUEUE_DOCKER_IMAGE. If anyone runs the stack against a non-Debian rabbitmq image (e.g. an alpine variant or a custom build) that lacks bash, the exec-form probe fails to find bash, the healthcheck reports unhealthy, and depends_on: service_healthy for message-queue blocks the whole stack from starting. The previous rabbitmq-diagnostics form worked on any rabbitmq image. Note the fragility in the comment or fall back to a form that doesn't depend on bash when /bin/sh is dash.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At python_testcontainers/infrahub_testcontainers/docker-compose-cluster.test.yml, line 40:

<comment>This healthcheck hard-requires bash in the broker image, but the image is configurable via MESSAGE_QUEUE_DOCKER_IMAGE. If anyone runs the stack against a non-Debian rabbitmq image (e.g. an alpine variant or a custom build) that lacks bash, the exec-form probe fails to find bash, the healthcheck reports unhealthy, and depends_on: service_healthy for message-queue blocks the whole stack from starting. The previous rabbitmq-diagnostics form worked on any rabbitmq image. Note the fragility in the comment or fall back to a form that doesn't depend on bash when /bin/sh is dash.</comment>

<file context>
@@ -34,7 +34,10 @@ services:
+      # raw TCP probe instead of rabbitmq-diagnostics: each diagnostics call
+      # boots a full Erlang VM (~2s CPU), which at a 1s interval pegs two
+      # cores per broker for the life of the stack
+      test: ["CMD", "bash", "-c", "</dev/tcp/127.0.0.1/5672"]
       interval: 1s
       timeout: 1s
</file context>

[ogenstad]

I'd say it looks good to me but this is awful. Assuming that the command is just a wasteful tcp check it looks fine with me but as I wrote I think we also need to change this everywhere.

[ogenstad]

I'm wondering if the problem here is rabbitmq-diagnostics -q check_port_connectivity or if it's the interval of 1 second. Looks like we have the same check with a 5 second interval in the regular setup (i.e. under ./development). If this truly is just a tcp handshake it seems like an awful waste to start an Erlang VM for it. 🤯 But we should not only change it for the devcontainers but for all of our rabbitmq health checks.