feat(external-api): round-trip dashboard containers, tabs, and tile container/tab refs

[alex-fedotyev]

Summary

PR #2015 added a dashboard organization layer (containers with optional tabs, plus per-tile containerId and tabId) but the v2 external API was not updated to round-trip the new fields. External integrations that build dashboards programmatically had no way to use the new layer.

This wires the full set of fields through CREATE / GET / LIST / UPDATE on /api/v2/dashboards. Dashboards saved without containers round-trip unchanged.

Closes #2150. Follow-up to #2015 (commit 7665fbe).

What's in scope

• Dashboard body Zod schema gains containers: DashboardContainer[]? (imported from @hyperdx/common-utils) and the tile schema gains containerId? and tabId?.
• convertToExternalDashboard now emits containers (only when at least one is present, so dashboards without the layer round-trip with the field absent).
• convertTileToExternalChart and convertToInternalTileConfig propagate containerId and tabId. The legacy series-format translator in externalApi.ts also propagates them so both code paths preserve the fields.
• The containers: 1 projection is added to the Mongoose find and findOne calls.
• New cross-field validation on the body schema:
  • container ids unique within a dashboard
  • tab ids unique within a container
  • tile containerId resolves to a real container
  • tile tabId resolves to a tab inside that container
  • tile tabId requires containerId to be set
• OpenAPI JSDoc additions for DashboardContainer, DashboardContainerTab, the new tile fields, and the new dashboard field on Dashboard / CreateDashboardRequest / UpdateDashboardRequest. openapi.json regenerated.
• A changeset entry.

Out of scope

Each item below has a tracking issue so the gap is visible after merge.

• The legacy type: 'section' discriminator removed in refactor: Unify section/group into single Group with collapsible/bordered options #2015 (commit 7665fbe). Not emitted, not validated against. Legacy documents parse cleanly via Zod's strip-unknown. No follow-up needed; documenting for clarity.
• Container-level repeat field (future work per refactor: Unify section/group into single Group with collapsible/bordered options #2015 review notes). Tracked in External Dashboards API: support container repeat field #2213.
• The MCP hyperdx_save_dashboard tool is unchanged in this PR. Its inline inputSchema does not yet expose containers, and adding it there is its own follow-up; doing both at once would tangle two surfaces. Tracked in External Dashboards API: expose containers and tabs in MCP hyperdx_save_dashboard #2212.
• Heatmap chart-type round-trip lands separately in [HDX-4120] feat(api): support heatmap tiles in external dashboards API #2200.
• Per-tile onClick drill-down round-trip is tracked in External Dashboards API: support per-tile onClick drill-down #2214.

Tier

The triage classifier marks packages/api/src/routers/external-api/v2/* as critical-path, so this lands as Tier 4 by directory rule, even though the diff is small (~284 prod lines) and additive. Splitting further would separate the body schema, the conversion utilities, and the route wiring from each other and not actually reduce review burden. Happy to break this up if there's a preferred way to slice it.

Test plan

• yarn ci:lint (lint + tsc + spectral) on @hyperdx/common-utils, @hyperdx/api, @hyperdx/app
• yarn knip (no new unused exports)
• Integration: yarn jest dashboards.test.ts -t "Containers and tabs", all 8 new tests pass
• Integration: full yarn jest dashboards.test.ts, 86/86 tests pass (no regressions in old or new format suites)
• Integration: yarn jest src/mcp/__tests__/dashboards.test.ts, 19/19 MCP dashboard tests pass (the MCP body schema shares with the external API body schema, so this confirms the new validations don't break the MCP path)
• openapi.json regenerated and committed; spectral lint passes

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
hyperdx-oss	Ready	Preview, Comment	May 8, 2026 2:24am

[changeset-bot]

🦋 Changeset detected

Latest commit: 815a063

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages

Name	Type
@hyperdx/api	Minor
@hyperdx/common-utils	Patch
@hyperdx/app	Minor
@hyperdx/otel-collector	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
hyperdx-oss	Building	Preview, Comment	May 5, 2026 8:19pm

[github-actions]

🔴 Tier 4 — Critical

Touches auth, data models, config, tasks, OTel pipeline, ClickHouse, or CI/CD.

Why this tier:

• Critical-path files (2):
  • packages/api/src/routers/external-api/v2/dashboards.ts
  • packages/api/src/routers/external-api/v2/utils/dashboards.ts
• Cross-layer change: touches backend (packages/api) + shared utils (packages/common-utils)

Review process: Deep review from a domain expert. Synchronous walkthrough may be required.
SLA: Schedule synchronous review within 2 business days.

Stats

• Production files changed: 6
• Production lines changed: 704 (+ 1059 in test files, excluded from tier calculation)
• Branch: alex/HDX-2150-external-api-containers-tabs
• Author: alex-fedotyev

To override this classification, remove the review/tier-4 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

[github-actions]

PR Review

✅ No critical issues found.

This is a well-scoped follow-up to #2015 that wires containers/tabs/tile containerId/tabId through the v2 external API. The implementation is careful and the test coverage (round-trip, validation, boundary, legacy-doc heal, PUT-omit-containers, orphan-heal) is thorough.

A few non-blocking observations for the team's consideration:

• Error format consistency — collectTileContainerRefIssues returns path: message strings joined with '; ' and surfaces them via { message }. Other handler-level 400s in this file use prose (Could not find the following source IDs: ...). Not wrong, but slightly inconsistent — fine to leave.
• PUT validates body against existing dashboard's containers when body omits containers (v2/dashboards.ts:1993-2003). This is correct per the documented preserve-on-omit semantics and is tested. Worth noting that the body schema only does structure checks (duplicate ids/tabs); cross-tile reference resolution depends on the handler. Anyone removing the handler-level collectTileContainerRefIssues call would silently weaken validation — a code comment near the schema-level structure check pointing at the handler would be defensive.
• Legacy duplicate container ids in stored docs — convertToExternalDashboard builds containerById last-write-wins (v2/utils/dashboards.ts:402-404). Self-heal logs orphan-ref drops but won't surface that the wrong container was matched if a legacy doc has duplicate ids. Acceptable since save paths now reject duplicates; pre-existing data only.
• TileSchema.containerId/tabId tightened to .min(1) in common-utils/types.ts. The external API normalizes empty strings to absent on read, so this is safe for legacy docs going through the v2 API. Worth verifying no UI save path persists empty strings (a quick grep shows nothing obvious).

The split into structure-only + tile-ref helpers (validateDashboardContainersStructure / validateDashboardTileContainerRefs) plus the collectTileContainerRefIssues wrapper handles the PUT-omit-containers case cleanly. Caps on identifiers/arrays (DASHBOARD_CONTAINER_ID_MAX, DASHBOARD_MAX_CONTAINERS, DASHBOARD_MAX_TILES) are sensible.

Out-of-scope items (MCP, heatmap, repeat, onClick) are tracked in linked issues per the PR description.

[github-actions]

E2E Test Results

✅ All tests passed • 165 passed • 3 skipped • 1162s

Status	Count
✅ Passed	165
❌ Failed	0
⚠️ Flaky	4
⏭️ Skipped	3

Tests ran across 4 shards in parallel.

View full report →

[github-actions]

Compound Engineering Review

Five specialist reviewers (TypeScript-quality, security, performance, architecture, simplicity) ran in parallel against the diff vs. base 2785597a. No P0s. Five P1s, mostly clustered around missing input caps, a missed projection cost, and an under-typed Mongoose update.

P1 — should resolve before merge

• P1 packages/api/src/utils/zod.ts:423-424 + :474 — tile.containerId/tabId have only min(1) (no .max) and externalDashboardTileListSchema has no .max on the array. With the 32MB body limit (api-app.ts:55) and tiles: Schema.Types.Mixed, a client can persist a huge tile list with multi-MB id strings that round-trip back through every GET → cap ids at DASHBOARD_CONTAINER_ID_MAX (256) to mirror DashboardContainerSchema, and add .max(N) (e.g. 500) to the tile list schema.
• P1 packages/api/src/routers/external-api/v2/dashboards.ts:1398 — LIST endpoint now selects containers for every dashboard. Worst case ≈310KB × 200 dashboards = ~62MB JSON per LIST → drop containers from EXTERNAL_DASHBOARD_PROJECTION and require GET-by-id to read it (or paginate LIST).
• P1 packages/api/src/routers/external-api/v2/dashboards.ts:1948 — setPayload: Record<string, unknown> opts out of Mongoose's UpdateQuery<IDashboard> typing; a typo like setPayload.contianers = … compiles and silently no-ops → type as mongoose.UpdateQuery<IDashboard>['$set'] (or Partial<IDashboard>); the conditional-assign idiom still works.
• P1 packages/common-utils/src/types.ts:1005-1015 — DashboardSchema.containers.refine(...) only checks duplicate container ids; tab-uniqueness and tile→container/tab resolution live only in the API-side superRefine. Internal write paths (DashboardWithoutIdSchema) and any future MCP write tool will accept dashboards the external API rejects → lift the four invariants into a shared validateDashboardLayout(data) helper in common-utils and call from both DashboardSchema.superRefine and buildDashboardBodySchema.
• P1 packages/api/src/routers/external-api/v2/utils/dashboards.ts:776 — Early return on hasDuplicateContainerId silences disjoint tile-level errors (unknown containerId, tabId requires containerId), forcing a two-step fix loop → drop the early return; if kept, gate only the tile→container resolution branch (the tabId requires containerId check is independent).

P2 — worth addressing

• P2 packages/common-utils/src/types.ts:899 — Tightening containerId/tabId to .min(1).optional() will reject any persisted tile with containerId: '' on next round-trip → run db.dashboards.find({"tiles.containerId":""}) before merge; if any rows exist, backfill or use .transform(s => s || undefined).
• P2 packages/api/src/models/dashboard.ts:36 — containers: Schema.Types.Array is free-form; defense-in-depth rests on Zod alone → declare a typed Mongoose subdoc mirroring DashboardContainerSchema, or add a Mongoose validator.
• P2 packages/api/src/routers/external-api/v2/dashboards.ts:39 — EXTERNAL_DASHBOARD_PROJECTION as const is not tied to IDashboard keys; dropping a model field leaves the projection silently stale → add satisfies mongoose.ProjectionType<IDashboard>.
• P2 packages/api/src/routers/external-api/v2/utils/dashboards.ts:760 — seenTabIds.add(tab.id) runs unconditionally, inconsistent with the container loop's else-branch placement; functionally identical but reads as a bug → move into the else.
• P2 packages/api/src/utils/externalApi.ts:36-208 vs v2/utils/dashboards.ts:303-313, 490-516 — Two parallel external↔internal tile translators each got containerId/tabId plumbed through; future tile-layout fields will need both touched → extract pickTileLayoutFields(tile) returning {id, x, y, w, h, name, containerId?, tabId?} and call from both.
• P2 packages/api/src/utils/externalApi.ts:206 — No test exercises the legacy series-format translator with containers/tabs → add one fixture in the legacy suite to lock parity.
• P2 packages/api/src/routers/external-api/v2/utils/dashboards.ts:806 — container.tabs?.find(t => t.id === tile.tabId) is O(T) per tile (T capped at 20 so OK now); since containerById is already built, attach a tabsById map per container during the first pass to make tile lookups O(1) → trivial restructure.
• P2 packages/api/src/routers/external-api/v2/dashboards.ts:1191-1195, 1242-1246, 1293-1297 — containers JSDoc duplicated verbatim across Dashboard/CreateDashboardRequest/UpdateDashboardRequest → extract a DashboardContainersField component and $ref it (mirrors what DashboardContainerTab already does).
• P2 packages/api/src/routers/external-api/v2/utils/dashboards.ts:786, 807 — Error messages echo client-supplied ids verbatim into 400 responses and Pino logs; with ids capped (see P1) the bloat risk reduces, but consider truncating echoed ids to ~64 chars regardless.

Verified non-issues

• NoSQL operator injection: Zod object schemas strip $set/$where/dotted-key payloads before they reach Mongoose.
• IDOR / authorization: every endpoint scopes by team: teamId; the new projection only adds containers: 1, no cross-tenant fields.
• Container-cap DoS math: 50 × 20 × ~512 chars ≈ 512KB worst case for containers itself, well under the 32MB body limit. The real fan-out risk is on tiles (P1 above), not the container cap.
• Scope discipline: 11 new integration tests cover the new contract; 5 deferred follow-ups ([HDX-4120] feat(api): support heatmap tiles in external dashboards API #2200, External Dashboards API: expose containers and tabs in MCP hyperdx_save_dashboard #2212-External Dashboards API: support per-tile onClick drill-down #2214) are tracked.

Reviewers run

kieran-typescript-reviewer, security-sentinel, performance-oracle, architecture-strategist, code-simplicity-reviewer.

[alex-fedotyev]

P2 items addressed in 574ee8e:

• Tightened internal TileSchema.containerId/tabId to min(1).optional()
• .max() bounds added: 256-char ids/titles, 20 tabs/container, 50 containers/dashboard. The external API caps containers so a client cannot trigger O(n*m) refine cost
• Single-pass over containers (container-id uniqueness + per-container tab-id uniqueness + map build), short-circuit if container-id duplicates exist before attempting tile resolution
• EXTERNAL_DASHBOARD_PROJECTION constant in v2/dashboards.ts so the GET-list and GET-by-id projections stay in sync
• Three missing tests added: PUT-path duplicate-container-id rejection, tile with containerId set when containers is omitted entirely (was a NPE-by-coincidence), tile in a tabbed container that omits tabId

Cross-schema invariant lifting + applyCommonTileFields deferred to follow-up #2225 (the lift would tighten DashboardSchema validation against existing internal writers, which needs its own scoped review and a production-data scan).

[github-actions]

Deep Review

✅ No critical issues found.

The PR has already been through several compound-engineering review rounds (commits "tighten container/tab invariants per compound-review", "deep-review followups", "preserve containers on PUT and self-heal orphan refs"). Cross-reviewer attention turned up no P0/P1 defects, but agent-surface parity and a concurrent-PUT silent-self-heal interaction are worth surfacing.

🟡 P2 — recommended

• packages/api/src/routers/external-api/v2/dashboards.ts:2041 — Two concurrent PUTs (one omitting containers, one explicitly []) leave Mongo with orphan tile→container refs because there is no updatedAt/__v guard, and the read-time self-heal in convertTileToExternalChart silently strips the orphan refs from the response so the data loss is invisible to clients.
  • Fix: Add an optimistic-concurrency filter on the findOneAndUpdate so the loser of a race gets 409, or surface a X-HyperDX-Healed-Refs response header listing tile ids whose refs were dropped.
  • _adversarial
• packages/api/src/mcp/tools/dashboards/saveDashboard.ts:42 — The MCP hyperdx_save_dashboard inputSchema does not expose containers, and the MCP update path does not preserve them like the v2 PUT does, so an agent calling the tool on a dashboard that already has containers silently flattens the layout — even though hyperdx_get_dashboard returns the new fields, breaking read/write parity.
  • Fix: Either preserve existingDashboard.containers on the MCP update path (mirroring v2 PUT) until External Dashboards API: expose containers and tabs in MCP hyperdx_save_dashboard #2212 lands, or note the silent-flatten risk in External Dashboards API: expose containers and tabs in MCP hyperdx_save_dashboard #2212 so the gap is closed before agents hit it.
  • _agent-native
• packages/cli/src/cli.tsx:984 — hdx dashboards --json (advertised as LLM/agent output) hand-picks fields and drops containers plus per-tile containerId/tabId, so CLI-driven agents see a flat tile list and lose grouping. Not tracked in any follow-up.
  • Fix: Add containers to the JSON projection, include containerId/tabId in the per-tile object, and update the help-text schema block at lines 962-977.
  • _agent-native

🔵 P3 nitpicks (6)

• packages/common-utils/src/types.ts:1142 — Exported validateDashboardContainersConsistency has no production caller; only the unit-test file references it. The two underlying primitives (validateDashboardContainersStructure and validateDashboardTileContainerRefs) are what production uses, so the composite reads as load-bearing but is dead.
  • Fix: Delete the composite and retarget its test at the two primitives, or wire the composite into a real call site.
  • _{maintainability}
• packages/common-utils/src/types.ts:1172 — Block comment claims cross-tile validation is applied via validateDashboardContainersConsistency at the external-API request body schema, but the schema actually invokes the two primitives directly; the named helper is unused.
  • Fix: Update the comment to point at validateDashboardContainersStructure + validateDashboardTileContainerRefs, or wire the composite where the comment claims it lives.
  • _{maintainability}
• packages/common-utils/src/types.ts:1186 — DashboardSchema.containers inlines a superRefine that duplicates the container-id uniqueness check already implemented in validateDashboardContainersStructure, so the two can drift silently and the inline copy uses a slightly different issue path ([i, 'id'] vs [...containersPath, i, 'id']).
  • Fix: Replace the inline superRefine with validateDashboardContainersStructure(containers, ctx, { containersPath: [] }) so internal and external schemas stay in lockstep.
  • _{maintainability, kieran-typescript}
• packages/api/src/routers/external-api/v2/utils/dashboards.ts:825 — collectTileContainerRefIssues builds z.object({}).superRefine(...).safeParse({}) purely to obtain a RefinementCtx, then string-formats the resulting issues; the empty-schema parse contributes nothing and the inline (_, ctx) => shadows the lodash _ import at the top of the file.
  • Fix: Extract a non-Zod issue collector that both this helper and validateDashboardContainersConsistency route through, returning string[] directly.
  • _{maintainability, kieran-typescript}
• packages/api/src/routers/external-api/v2/dashboards.ts:1993 — On a PUT that omits containers, the handler falls back to existingDashboard.containers without running validateDashboardContainersStructure, and collectTileContainerRefIssues builds its lookup map via containers.map(c => [c.id, c]) (last-write-wins on duplicates); a legacy doc with duplicate container ids produces a misleading 'unknown tabId' 400 with no recovery path through the API.
  • Fix: Run the structure validator on effectiveContainers when sourced from the existing doc and short-circuit on hasDuplicateContainerId, surfacing a clear server-data-corruption error instead.
  • _{correctness, adversarial}
• packages/api/src/routers/external-api/v2/dashboards.ts:1707 — Handler-level tile-ref errors omit the Body validation failed: prefix that body-schema errors carry, so POST/PUT clients see two different error envelopes for what is conceptually one validation class.
  • Fix: Prefix the joined tile-ref message with Body validation failed: so the envelope is consistent across the schema and handler validation paths.
  • _correctness

---

Reviewers (10): correctness, testing, maintainability, project-standards, agent-native, learnings-researcher, security, api-contract, adversarial, kieran-typescript.

Testing gaps:

• No test exercises a PUT with explicit containers: [] against a dashboard that already has containers — only POST with [] is covered, leaving the if (containers !== undefined) branch in the PUT handler untested for the non-empty→empty transition.
• Legacy series-format translator (packages/api/src/utils/externalApi.ts) now passes through containerId/tabId, but no integration test exercises this path.
• Same tab.id reused across two different containers (which the per-container uniqueness scope implies should be allowed) is not pinned by a test, so a future scope change would not be caught.
• DASHBOARD_MAX_CONTAINERS = 50 and DASHBOARD_CONTAINER_MAX_TABS = 20 boundaries are unenforced by tests; only the 500-tile cap is exercised.
• A tile in container A whose tabId matches a tab only in container B is not distinguished from the generic 'unknown tabId' case.
• No test covers the concurrent-PUT race or the read-time self-heal observability (orphan refs persisted via direct DB mutation).