Skip to content

demo/grafana: pure-wire latency panel + clearer latency info tips (both boards) - #24

Open
armcconnell wants to merge 1 commit into
mainfrom
docs/dashboard-wire-latency-panel
Open

demo/grafana: pure-wire latency panel + clearer latency info tips (both boards)#24
armcconnell wants to merge 1 commit into
mainfrom
docs/dashboard-wire-latency-panel

Conversation

@armcconnell

Copy link
Copy Markdown
Contributor

Summary

Adds a Wire latency (publisher→recv, network only) timeseries panel (p50/p95/p99 of send_latency_ms) to both dashboards, and sharpens the latency panel descriptions.

Motivation: source→recv is dominated by the venue→publisher ingest lag (block→emit), which can tail to seconds and is not network. send→recv (recv_ts − publisher_send_ts, kernel-stamped via SO_TIMESTAMPNS) isolates pure transport. The new panel surfaces that directly.

Confirmed live: TOB wire ≈ 68/69/70 ms (flat); MBO wire ≈ 78/210/274 ms (the p99 bump is publisher-side burst serialization at block boundaries) — both far below the source→recv ingest figures.

Changes

  • New wire-latency panel on topofbook and marketbyorder boards (identical except the source table).
  • Info tips: send→recv = pure wire; source→recv = wire + block→emit ingest lag; quantiles panel explains the send/source gap. Filled in previously-empty descriptions too.
  • Kept the two boards as close to identical as possible.

Docs/dashboard-only; no code or schema change.

… (both boards)

Adds a 'Wire latency (publisher->recv, network only)' timeseries (p50/p95/p99
of send_latency_ms) to both the Top-of-Book and Market-by-Order dashboards, so
the network/transport number is separated from the venue->publisher ingest lag
that dominates source->recv.

Also fills in/clarifies panel descriptions, with the latency tips made explicit:
send->recv = pure wire (kernel-stamped recv), source->recv = wire + block->emit
ingest lag. Mirrored identically across both boards.
@juan-malbeclabs
juan-malbeclabs force-pushed the docs/dashboard-wire-latency-panel branch from 136a3d0 to 731d5de Compare September 1, 2026 23:52
@juan-malbeclabs

Copy link
Copy Markdown
Contributor

Rebasado sobre main (sin conflictos, sin cambios en el contenido del PR). El motivo: los jobs de Rust dejaron de estar filtrados por paths en #62, así que una rama anterior a eso lleva el workflow viejo y ninguno de ellos reporta — se ve en los checks de este PR antes del rebase. Eso importa ahora porque el siguiente paso es agregarlos a los checks requeridos del ruleset, y un check requerido que nunca reporta bloquea el PR para siempre.

Si tenías la rama local, hace falta git fetch && git reset --hard origin/docs/dashboard-wire-latency-panel.

@juan-malbeclabs juan-malbeclabs left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. What I could check from the repository, I did.

Both dashboards parse. The new panel is present on each, panel ids are unique within each board, and no two panels overlap in gridPos — 22 panels on topofbook, 23 on marketbyorder, zero collisions. A panel dropped on top of another is the usual cost of hand-edited dashboard JSON and it is not here.

The part that would have made the panel useless: send_latency_ms exists as a MATERIALIZED column — (toUnixTimestamp64Nano(recv_ts) - toUnixTimestamp64Nano(publisher_send_ts)) / 1e6 — on both tables the panel actually queries, topofbook.quotes and marketbyorder.events. So the query resolves against the schema in this repository rather than against a column that only exists on one deployment.

The framing is right and worth having: source→recv is dominated by venue→publisher ingest and is not a network measurement, so a board that offers only that invites an operator to read block→emit lag as transport. Isolating send→recv is the fix.

One thing a reader should know about the descriptions. The figures and the causal claim — TOB ≈ 68/69/70 ms, MBO ≈ 78/210/274, and "the p99 bump is publisher-side burst serialization at block boundaries" — come from a live run nobody else can reproduce from this repository. The interpretation may well be right; it is not something the panel's own data proves, and it is now printed in the panel description where it reads as established. Not a reason to hold a dashboard change, and worth knowing before somebody cites it back at a publisher.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants