The backend exports metrics, traces and logs over OTLP. Endpoint, protocol and
headers come from the standard OTEL_EXPORTER_OTLP_* environment variables, so
nothing is configured in code. Dashboards live in grafana/dashboards/ and are
Grafana schema v2 documents.
Connection recovery and correlated diagnostics are covered in connection-recovery.md. Importable metrics alert rules live in prometheus/connection-alerts.yml.
| Dashboard | Answers |
|---|---|
flightstrips-backend.json |
Are connections, messages and PDC healthy? |
flightstrips-traffic.json |
What is the traffic situation at the airport? |
flightstrips-usage.json |
Who is connected, and for how long? |
flightstrips-performance.json |
Why is the backend busy? |
flightstrips-aman.json |
Did a prediction spike, STAR exchange, promotion or spacing change move arrival guidance? |
Run npm run test:contract:aman in frontend/ with the repository's Go and
Node toolchains installed. It generates current JSON using the Go AMAN
publisher and sends it through the actual TypeScript consumer, WebSocket event
handlers, Zustand store and rendered AMAN route, without mocking UI components.
The dedicated AMAN publisher-consumer contract CI check runs on every pull
request and merge queue entry, including changes confined to either service.
It also runs the full AMAN backend suite, frontend tests and production frontend
build. This check is required in the main branch ruleset, so failures block merge.
Keep new AMAN behaviours represented in the generated scenarios and workspace
smoke tests when adding features or changing contracts.
The corpus includes healthy, filtered, unreliable, recovered, stale-source,
passed-feeder, degraded-weather and empty-airport states. The 26 October 2026
incident observations for CCA877Q are retained in
backend/pkg/events/frontend/testdata/aman-filter-2026-10-26.json. Replaying
them produces a traffic_prediction warning with component prediction and
code prediction_unreliable. A consumer must accept this warning while
retaining slots and healthy neighbouring guidance. Rejecting it discarded the
whole state, producing invalid_aman_state and hiding both AMAN and TMT.
Workspace checks also cover current warnings, visible targets, all twelve TMT
buckets, command revisions, observer/disconnect restrictions, stale replacements,
reconnect and recovery after a rejected replacement.
An aircraft's missing input or rejected calculation is local to that aircraft. It must not change the shared predictor's technical health or disable other aircraft's commands. Retained estimates and slots remain visible with stale data status. A prediction filter that cannot establish a consistent estimate withholds only that aircraft's live gain/loss; it retains the runway reservation. The flight-detail view carries the calculation's degradation reason.
Unreadable warning metadata is isolated from valid operational state. The frontend retains readable warnings and adds a visible notice about unavailable diagnostics; it still rejects malformed flight timings, health and envelope fields. The strict publisher/consumer validator continues to fail the required PR check for incompatible warning output, even though the live UI can recover.
Airport-wide technical readiness covers shared prerequisites such as current observations, navigation and persistence. A genuine source outage can restrict commands while the latest complete AMAN and TMT views remain visible. Supported weather fallback does not block authority. The required PR check exercises both aircraft-local degradation and an actual airport source outage.
EuroScope consumes a separate protobuf gain/loss replacement without the web warning list. An aircraft with unavailable guidance has all four presentation fields absent; other aircraft retain their complete timing fields. Stale inputs retain the last timing with stale status. Go wire tests and the plugin decoder regression cover mixed healthy, stale and unavailable aircraft in one message.
For the 26 October incident, CCA877Q's candidate ETA moved over three hours while its route distance changed from about 3,847 to 3,809 NM. The filter kept inconsistent candidates out and accepted a consistent pair at 19:06 CET. The incident fixture includes those recovery observations. This is evidence of a large changed estimate, not an airport-wide outage. The earlier accepted prediction's complete speed/model inputs were not logged, so the initiating model change cannot be conclusively reconstructed from these logs.
Filter logs now include current/previous distance, accepted/candidate model speed bases and whether the aircraft is actually unreliable. Committed guidance logs include the aircraft scope and degradation reason; calculation-error logs include session and aircraft scope.
AMAN STAR resolution evaluated records the reported assigned STAR, resolved
family, selected runway group and resolution outcome. A procedure recognized
on another configured runway can supply its family's path for the selected
runway without changing the reported assignment. Such accepted forecasts stay
fresh and carry the star_runway_inferred warning until the assignment matches.
Unknown or ambiguous procedures and families without a path for the selected
runway retain the previous estimate as stale. Pending runway rebuilds survive
navigation outages and restarts and finish on the next usable calculation.
The AMAN Stability dashboard separates automatic calculation attempts from committed target and guidance changes. Filter logs by airport, aircraft callsign regex and numeric session ID; its metrics are backend-wide. The spacing panel contains the demand counts and activation state used for each aircraft.
AMAN prediction filter evaluated records filtered candidates and automatically
accepted sustained changes. AMAN guidance change committed records before/after
physical and operational ETAs, target, guidance and neighbouring aircraft.
AMAN sequencing operation committed carries the actual optimizer, STAR swap,
gap-fill and vacancy-promotion operations. Join committed events by session and
revision, and controller actions by command ID. Failed commits and duplicate
commands do not emit committed-change events. The older ten-minute prediction
warning is a calculation attempt, not evidence of a committed target change.
Deploy backend support before updating EuroScope plugins. New plugins send
reported groundspeed and radar position age; older plugins use motion derived
over at least thirty seconds. Automatic filtering is enabled by default.
AMAN_PREDICTION_FILTERING_ENABLED=false disables the outlier filter for rollback
without removing diagnostics or native-motion support. It never asks controllers
to confirm predictions. Changes over three minutes need distinct observations
spanning thirty seconds and agreeing within a minute; two minutes of unresolved
instability suppresses actionable guidance while retaining the runway slot.
After rollout, inspect a busy EKCH session for isolated candidates that leave targets unchanged, automatically accepted sustained speed changes, and immediate promotions when genuine capacity becomes available. Compare committed changes, not raw warning counts, with the 19 October investigation.
The performance dashboard is meant to be read top to bottom. Each row narrows the question rather than repeating it.
Use the host metric for the saturation decision. The node exporter is already present in Grafana, so the dashboard uses the idle time across all host CPUs:
1 - avg(rate(node_cpu_seconds_total{job="integrations/node_exporter",mode="idle"}[$__rate_interval]))
The Go go.cpu.time classes are runtime attribution estimates and can be
useful for relative diagnostics, but they are not directly comparable to host
CPU time and must not be presented as server CPU utilisation.
Then read the class split:
userdominates — application code is the cost. Continue to row 2.gcis a large share — the process is allocating faster than it works. Cross-checkgo.gc.cyclesand heap size; the fix is allocation, not throughput.scavengeis visible — memory is being returned to the operating system under pressure, usually a symptom of a heap that has just shrunk sharply.
go.schedule.latency is the saturation signal. It measures how long a runnable
goroutine waited for a processor, so it rises before throughput falls. High
scheduling delay with low CPU utilisation means goroutines are blocked, not busy
— check go.mutex.wait.time for lock contention.
The Work Amplification row exists because the expensive failures in this system have not been slow code — they have been correct code running far more often than it needed to.
euroscope.sync.outcomesseparates syncs that changed persisted state from heartbeats that did not. EuroScope syncs on a fixed cadence, so a high unchanged rate is normal by itself.euroscope.sync.follow_up_workcounts what each sync queued for its finalization phase, by kind. This is the amplification factor between one inbound message and the work it causes.- Recalculations per unchanged sync is the ratio of the two. A sync that changed nothing should not need a full airport recalculation, so a non-zero value here is repeated work regardless of how fast each recalculation is.
euroscope.sync.phase.durationattributes a slow sync to one stage. The phases sum to the total, so the tallest line is the one worth profiling.cdm.recalculation.durationsummed as a rate gives processor-seconds per second: a value of 1 means one processor did nothing but CDM recalculation.
Both hubs dispatch from a single goroutine, so everything they send is serialised behind one loop.
websocket.hub.queue.depthis the backlog that loop is working through. Both queues hold 256 messages; a p99 approaching that means the hub can no longer keep up with its publishers.websocket.hub.broadcast.fanoutexplains dispatch cost growth without any code change — more clients in a session is more work per message.websocket.hub.publish.blockedcounts publishers that had to wait for room in the queue. This is measured only when the queue was actually full, so any sustained rate means back-pressure is now stalling the code producing events.websocket.clients.slow_disconnectscounts clients dropped for failing to drain their own send queue.websocket.message.bytes.receivedandwebsocket.message.bytes.sentcount serialized application payload bytes received from clients and accepted by the WebSocket writer.websocket.message.size.bytesrecords the corresponding payload-size distribution. All three use only the boundeddirection,source, andtypelabels. These measurements include the JSON or protobuf payload itself, but exclude WebSocket frame headers, TLS overhead, and any reduction from WebSocket compression.
pgxpool.acquired_connections against pgxpool.max_connections shows pool
saturation, and pgxpool.empty_acquire counts acquires that had to wait for a
connection. A saturated pool turns database latency into application queueing,
which presents as a slow backend rather than a slow database.
Individual query spans come from otelpgx and are enabled whenever
OTEL_EXPORTER_OTLP_ENDPOINT is set.
Only the session's master EuroScope client sends aircraft-position events. The
plugin checks this both when flushing positions and at the WebSocket send
boundary; the backend ignores position events from other clients. Direct-to and
assigned-speed facts instead come from the aircraft's tracking controller,
which may be a slave. Live TopSky /HOLD/, /XHOLD/, and /HOLD_EAT/
scratch-pad commands are broadcast to every operational controller, so the
session master publishes them once. Tracking controllers may reconcile active
holds from annotation 6, but a missing annotation never clears stored state.
Persisted hold fields are included in backend sync so a newly elected master
can apply a later standalone EAT. Commands observed during a backend outage are
replayed after reconnect. The backend accepts holding reports from either
authority. Hold fields in
full-sync and strip-update snapshots continue to follow tracking ownership so a
non-tracking snapshot cannot replace or clear the stored hold. Deploy the
backend and plugin changes together.
aircraft_position_update reads a typed strip/stand-assignment snapshot and
persists position, bay and euroscope_seen_at synchronously. The routine budget
is two database operations, or three with an established AMAN identity.
Unchanged stand-conflict evaluation may add one read. Genuine transitions,
identity changes and optimistic retries have separate budgets. Routine movement
inside the same routing region does not reload route state; failed refreshes
remain pending for a later report.
The Performance dashboard separates completion latency (socket receipt through
persistence and synchronous effects), queue delay, completed messages/sec and
sampled queue depth. Existing handler-duration sums measure accumulated wall
seconds across handlers; concurrent handlers can accumulate more than one
second per second. They are neither per-message latency nor CPU utilisation.
DB operations count actual pgx calls, including transactions and synchronous
publication reads. Traces use OTEL_SERVICE_VERSION, or embedded git revision
with a dirty suffix when available, instead of a fixed release string.
Lifecycle processing remains sequential until the concurrency load gates in
the position performance runbook pass.
Position database work is batched by default; set
POSITION_DB_BATCHING_ENABLED=false to restore individual statements.
POSITION_CONCURRENCY_ENABLED=true selects four workers per connection;
POSITION_WORKERS_PER_CLIENT=1 restores sequential dispatch while retaining DB
improvements. The shared position limit is eight, reduced for pool headroom.
Every dedicated report is processed without coalescing, with per-aircraft FIFO,
a 256-pending-report connection limit, while non-position operational messages
bypass the position dispatcher. Aircraft disconnects drain accepted positions
first to preserve lifecycle ordering.
Failed handler samples carry the bounded error_class label, including
serialization_conflict, deadlock, missing_row, and coordination, so
terminal failures can be counted without parsing log messages. Successfully
retried serialization conflicts and deadlocks are counted separately by
websocket.message.db_retries.
PDC request and clearance lifecycle logs include the aircraft callsign, session
and channel (CPDLC or WEB). Request failures include the available requested
and expected airports and aircraft type, plus the strip's flight-plan and clearance
state. Pending requests record outcome, pending_state, validation faults and
whether the pilot supplied remarks. Issuance failures identify the failing stage.
PDC Hoppie: Message accepted means Hoppie returned a successful application
response; pilot receipt is established separately by Pilot acknowledged clearance.
Sends log from, to, message_type, message_sequence and response_to when
available. Invalid HTTP responses, provider rejections and malformed poll responses
are errors. Diagnostic responses are bounded and the Hoppie logon code is redacted.
Callsigns are structured log metadata, so filter the fields rather than searching only the message text. Include both the reported callsign and a suspected stale callsign when investigating an aircraft logon mismatch:
{service_name="flightstrips-backend",deployment_environment="production"}
| callsign=~"AUA996|AUA998" or from=~"AUA996|AUA998" or to=~"AUA996|AUA998"
Full EuroScope syncs collect holding-clearance observations and apply them in one AMAN commit and publication per destination airport, rather than once per changed flight. Unchanged batches load the airport state once and do not commit or publish. Revision conflicts reload and reapply the batch, retaining newer concurrent clearances. A partial strip-sync failure still flushes observations for the strip writes that already succeeded.
Each changed AMAN projection uses one bulk flight upsert statement inside the existing transaction. Timeline and holding-EAT projections share one navigation geometry snapshot per publication; the next publication reads a fresh snapshot.
euroscope.sync.db_operations and the strip_update and
aircraft_position_update samples in websocket.message.db_operations count
actual pgx query calls during the handler, including nested AMAN queries and
transaction commands. Detached work after the handler completes is excluded.
Syncs retain manual accounting when no pgx tracer is installed. The expanded
coverage means these counts are not directly comparable with older sync and
strip-update metrics, which omitted downstream work. Pool acquires remain a
separate measure: multiple statements inside a transaction share one acquire.
For verification, compare LIVE sync/strip-update duration and pool acquire rate
at similar traffic levels. In a full-sync trace, holding changes should produce
at most one UpsertAMANFlights per affected airport without a revision retry,
and no sequence of individual UpsertAMANFlight calls.
Spans exist for the WebSocket upgrade, every WebSocket message, EuroScope sync, CDM recalculation, and every database query. The sync span carries the same phase breakdown and follow-up work counts as the metrics, as attributes, so one trace explains a single slow sync while the dashboard explains the trend.
Logs carry trace_id and span_id as structured metadata, so a warning in Loki
links directly to the trace that produced it.
Sampling is the SDK default of "sample everything", which is fine at current volumes. It is configurable without a code change through the standard environment variables when it stops being fine:
OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.1The dashboard identifies which path is expensive. When the answer needed is
which function, the backend serves net/http/pprof on 127.0.0.1:6060. It is
off by default and bound to loopback, so it is reachable only from inside the
container:
docker compose -f docker-compose.prod.yml exec backend wget -qO /tmp/cpu.pprof http://127.0.0.1:6060/debug/pprof/profile?seconds=30Enable it with ENABLE_PPROF=true and restart the backend.
Metric attributes use a closed vocabulary. Callsigns, CIDs, route text, command
identifiers and provider payloads are never metric dimensions — they belong in
span attributes and log fields, where high cardinality is free. fixedLabel in
backend/internal/metrics/metrics.go collapses any unexpected value to other
so a new enum value can never multiply a time series.