Skip to content

Commit e488794

Browse files
nan-yuzeroasteriskgemini-code-assist[bot]
authored
feat(composer): A2UI Theater — interactive JSONL playback and streaming viewer (google#987)
* docs: design system integration guide, custom components guide, YouTube component - New guide: design-system-integration.md — step-by-step for adding A2UI to an existing Material Angular application - Rewritten guide: custom-components.md — complete walkthrough for YouTube, Maps, and Charts custom components (replaces TODO skeleton) - New sample component: YouTube embed for rizzcharts catalog - Updated rizzcharts catalog.ts to include YouTube component - Friction log documenting 8 friction points (P2/P3) encountered during development, with recommendations - Added Design System Integration to mkdocs nav * Apply suggestions from code review Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * docs: address PR google#824 review comments - Remove friction log file (content already in issue google#825) - YouTube component: add video ID regex validation (security) - custom-components.md: rename to 'Custom Component Catalogs', reorder examples (media first), clarify basic catalog is optional, remove redundant heading, fix Maps input.required consistency, add encodeURIComponent to docs example - design-system-integration.md: rewrite to focus on wrapping Material components as A2UI components (not using DEFAULT_CATALOG), show custom catalog without basic components, add mixed catalog example - s/standard/basic/ throughout * docs: add render_macros:false to prevent Jinja2 eval of Angular template syntax * feat(dojo): Add A2UI Dojo and Mock Scenarios * feat(dojo): implement comprehensive visual design and layout polish for A2UI Dojo - Redesigned Top Command Center with glassmorphic header and functional timeline scrubber. - Replaced native scenario select with Shadcn DropdownMenu. - Polished Data Stream view with active state highlighting, glow effects, and auto-scrolling. - Replaced native checkboxes with custom Tailwind styled toggles in Config view. - Added dynamic grid layout for the Renderers Panel with sophisticated styling per surface (React Web, Discord dark mode replica, Lit Components). - Applied custom slim scrollbars throughout for a premium feel. * fix(composer): remove edge runtime to fix Next.js build errors, prefer Vercel over Cloudflare Pages * feat(dojo): Add scenario harvest, mobile layout, and UX evaluations * fix(composer): remove opennextjs-cloudflare dependency for Vercel deployment * fix(dojo): fix progress timeline and start renderers in empty state * feat(dojo): enable URL parameter deep linking for scenarios and timeline steps * feat: wire up A2UIViewer to dojo scrubber stream via useA2UISurface hook * feat(dojo): polish timeline scrubber, sync A2UI transcoder and adjust layout * feat(dojo): fix A2UISurface hook compatibility for React renderer and sync sample data * feat(dojo): improve visual feedback for timeline and scrubber messages * feat(dojo): integrate northstar-tour scenario, remove CopilotKit, and update transcoder * fix(dojo): use real v0.8 sample scenarios and fix renderer pipeline - Replace invented v0.9 scenarios with real v0.8 samples from samples/agent/adk/ - Add restaurant-booking, restaurant-list, restaurant-grid, restaurant-confirmation - Add contact-card, contact-list, org-chart, floor-plan scenarios - Update index.ts to surface real scenarios as defaults - Default scenario is now restaurant-booking (verified rendering) - Fix transcoder.ts: pass v0.8 messages through unchanged - Fix useA2UISurface.ts: only process v0.8 format components (id + component) - Fix dataModelUpdate: parse ValueMap format correctly - Restaurant booking now renders: column, text, image, textfield, datetime, button - Locally verified with headless Chromium: all A2UI CSS classes present - Build passes (bun run build) * fix(dojo): use @copilotkit/a2ui-renderer instead of local @a2ui/react - Switch import to @copilotkit/a2ui-renderer (npm-published) - Remove file:../../renderers/react dep that breaks Vercel builds - @copilotkit/a2ui-renderer uses @a2ui/lit under the hood (npm transitive dep) * feat(dojo): single renderer pane, step summaries, hide Discord mock - Remove Discord mock pane and multi-renderer grid - Show single A2UI renderer (full width, centered) - Add human-readable step summaries to JSONL pane (e.g. 'Update 9 components: Column, Text, Image, TextField...') - Raw JSON collapsed by default behind 'Raw JSON ▸' toggle - Steps are clickable to seek directly - Wider left pane (35%) for better readability - Remove unused renderer toggle state * fix(dojo): remove broken v0.9 scenarios that crash renderer - Remove northstar-tour, flight-status, weather-widget (v0.9 format) - Remove kitchen-sink, component-gallery-stream (not real scenarios) - Keep only verified v0.8 scenarios from samples/agent/adk/ - 12 working scenarios remain in dropdown * fix(dojo): use standard catalog for rizzcharts-chart scenario so it renders * fix(dojo): remove h-full overflow-hidden from renderer frame so content doesn't clip The browser-chrome container had h-full which constrained it to viewport height and overflow-hidden clipped any content below. This caused the DateTimeInput and fields below it to be cut off in restaurant-booking with no way to scroll. Fix: remove h-full from the frame so it grows with content, and drop flex-1 + overflow-auto from the content area since the outer pane wrapper already handles scrolling via overflow-y-auto. Fixes google#243 * fix(dojo): add missing confirmation-column component to restaurant-confirmation scenario The Card component referenced 'confirmation-column' as its child, but that component was never defined in the surfaceUpdate. This caused the model processor to return null for the unresolved child, making isResolvedCard() fail validation with 'Invalid data; expected Card'. Fix: add the confirmation-column Column component with an explicit children list wiring up all the confirmation content components. * dojo: hide mock browser chrome traffic light dots on mobile viewport On small screens (< sm breakpoint), the traffic light dots in the mock browser chrome header are hidden to reduce visual clutter. Uses Tailwind's hidden sm:flex responsive pattern. Task google#237 * fix(dojo): set copilotkit route to force-dynamic to fix 500 on page load The route was incorrectly set to 'force-static' which caused Next.js to fail with a 500 error on every page load since the CopilotRuntime / InMemoryAgentRunner cannot be statically exported. Change to 'force-dynamic' so the route is properly handled server-side. * feat(dojo): URL state, config panel, nav link, curated scenarios - URL state: scenario, step, renderer synced to query params (e.g. /dojo?scenario=contact-card&step=2&renderer=React) - Config panel: renderer dropdown + scenario dropdown (synced with header) - Sidebar nav: added 'Dojo' link with Play icon - Curated to 5 quality scenarios (removed broken/redundant ones) - Removed contact-lookup (crashes on step 4 — missing component refs) * fix: use npm lockfile for Vercel compatibility * fix: gitignore pnpm-lock.yaml to prevent Vercel from using pnpm * feat(dojo): streaming simulation, 3-tab left pane, remove header scenario Major architecture change: - NEW useStreamingPlayer hook: explodes messages into individual JSONL lines that stream progressively (line-by-line) instead of whole-message chunks - Scrubber now has fine-grained control over 60+ stream positions per scenario - Three left pane tabs: (a) Events — lifecycle summaries (surface created, components registered, data bound) (b) Data — raw streaming lines appearing chunk by chunk with ↓/↑ badges (c) Config — scenario, renderer, transport dropdowns - Removed scenario dropdown from header (lives only in Config tab) - Streaming cursor animation shows which message is mid-delivery - Click any event/line to seek to that position - Server/client sections grouped with direction badges - Compact header with streaming status indicator * fix(dojo): correct JSONL streaming — chunk per message, not per line Each JSONL chunk is one complete JSON object on one line — that's how real SSE/JSONL works. Not individual formatted lines within a message. - Data tab now shows raw wire format (compact JSON, one chunk per card) - Shows byte size per chunk and total bytes received - Scrubber steps through 3 chunks for restaurant-booking (not 60+ fake lines) - Streaming cursor shows 'Waiting for next chunk...' between deliveries - Header shows total bytes received during playback * feat(dojo): add user interaction to restaurant-booking scenario Adds 3 new chunks to the restaurant-booking scenario: - Chunk 4 (↑ CLIENT): User submits booking form with party size, date, dietary requirements via clientEvent/userAction - Chunk 5 (↓ SERVER): Agent responds with confirmation surface update (checkmark, title, details, summary, Done button) - Chunk 6 (↓ SERVER): Data model update with confirmation details Full bidirectional flow: server renders form → user fills and submits → server confirms with new surface state. Demonstrates the complete A2UI interaction lifecycle in 6 JSONL chunks. * Use relative paths for agent output * Update record_scenario script to recording streaming chunks * Automate image asset serving for streaming preview * Skip copying if the image already exists in the destination * Update recorded scenarios with streaming chunks * Disable non-streaming scenarios * Fix CI failures * Add licenses to composer * chore: migrate package manager from npm to pnpm in composer tool It fixes compose CI build failure: https://github.com/google/A2UI/actions/runs/23568752574/job/68626383193?pr=987. ``` /home/runner/setup-pnpm/node_modules/.bin/pnpm store path --silent /home/runner/setup-pnpm/node_modules/.bin/store/v10 Error: Some specified paths were not resolved, unable to cache dependencies. ``` * feat: use @a2ui/markdown-it as optional peer dependency for v0.8 in Lit and Angular * Update streaming chunks * refactor(composer): rename Dojo → Theater, add keyboard shortcuts, remove deployment artifacts - Rename /dojo route to /theater throughout (app, components, data, nav) - Add keyboard shortcuts: Space (play/pause), Arrow keys (step ±1), Shift+Arrow (step ±5) - Remove deployment-specific files: open-next.config.ts, wrangler.jsonc, .dev.vars, _headers Co-authored-by: Nan Yu <[email protected]> * Remove unrelated doc update * refactor: remove debug console log from A2UIRenderer * refactor: remove obsolete patch_v2 script for page.tsx step support --------- Co-authored-by: alan blount <[email protected]> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
1 parent 3b1dc2d commit e488794

68 files changed

Lines changed: 30705 additions & 3103 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

renderers/angular/package.json

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,13 @@
3636
"peerDependencies": {
3737
"@angular/common": "^21.2.0",
3838
"@angular/core": "^21.2.0",
39-
"@angular/platform-browser": "^21.2.0"
39+
"@angular/platform-browser": "^21.2.0",
40+
"@a2ui/markdown-it": "file:../markdown/markdown-it"
41+
},
42+
"peerDependenciesMeta": {
43+
"@a2ui/markdown-it": {
44+
"optional": true
45+
}
4046
},
4147
"devDependencies": {
4248
"@angular/build": "^21.2.0",

renderers/angular/src/v0_8/data/markdown.ts

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -28,12 +28,24 @@ export abstract class MarkdownRenderer {
2828
providedIn: 'root',
2929
})
3030
export class DefaultMarkdownRenderer extends MarkdownRenderer {
31+
private static warningLogged = false;
32+
3133
override async render(
3234
markdown: string,
3335
options?: Types.MarkdownRendererOptions,
3436
): Promise<string> {
35-
// Basic implementation for v0.8
36-
return markdown;
37+
try {
38+
// @ts-ignore - optional peer dependency
39+
const { renderMarkdown } = await import('@a2ui/markdown-it');
40+
return await renderMarkdown(markdown, options);
41+
} catch (e) {
42+
if (!DefaultMarkdownRenderer.warningLogged) {
43+
console.warn("[DefaultMarkdownRenderer] Failed to load optional `@a2ui/markdown-it` renderer. Using fallback regex.");
44+
DefaultMarkdownRenderer.warningLogged = true;
45+
}
46+
// Basic implementation for v0.8
47+
return markdown;
48+
}
3749
}
3850
}
3951

renderers/lit/package.json

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -102,6 +102,14 @@
102102
"typescript": "^5.8.3",
103103
"wireit": "^0.15.0-pre.2"
104104
},
105+
"peerDependencies": {
106+
"@a2ui/markdown-it": "file:../markdown/markdown-it"
107+
},
108+
"peerDependenciesMeta": {
109+
"@a2ui/markdown-it": {
110+
"optional": true
111+
}
112+
},
105113
"dependencies": {
106114
"@a2ui/web_core": "file:../web_core",
107115
"@lit-labs/signals": "^0.1.3",
@@ -110,4 +118,4 @@
110118
"signal-utils": "^0.21.1",
111119
"zod": "^3.25.76"
112120
}
113-
}
121+
}

renderers/lit/src/0.8/ui/directives/markdown.ts

Lines changed: 16 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -61,13 +61,22 @@ class MarkdownDirective extends Directive {
6161
return until(rendered, html`<span class="no-markdown-renderer">${value}</span>`);
6262
}
6363

64-
if (!MarkdownDirective.defaultMarkdownWarningLogged) {
65-
console.warn("[MarkdownDirective]",
66-
"can't render markdown because no markdown renderer is configured.\n",
67-
"Use `@a2ui/markdown-it`, or your own markdown renderer.");
68-
MarkdownDirective.defaultMarkdownWarningLogged = true;
69-
}
70-
return html`<span class="no-markdown-renderer">${value}</span>`;
64+
const dynamicRendererPromise = (async () => {
65+
try {
66+
// @ts-ignore - optional peer dependency
67+
const { renderMarkdown } = await import('@a2ui/markdown-it');
68+
const rendered = await renderMarkdown(value, markdownOptions);
69+
return unsafeHTML(rendered);
70+
} catch (e) {
71+
if (!MarkdownDirective.defaultMarkdownWarningLogged) {
72+
console.warn("[MarkdownDirective] Failed to load optional `@a2ui/markdown-it` renderer. Using fallback regex.");
73+
MarkdownDirective.defaultMarkdownWarningLogged = true;
74+
}
75+
return html`<span class="no-markdown-renderer">${value}</span>`;
76+
}
77+
})();
78+
79+
return until(dynamicRendererPromise, html`<span class="no-markdown-renderer">${value}</span>`);
7180
}
7281
}
7382

renderers/web_core/package-lock.json

Lines changed: 2 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

renderers/web_core/package.json

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -119,7 +119,7 @@
119119
"google-artifactregistry-auth": "^3.5.0",
120120
"gts": "^7.0.0",
121121
"rxjs": "^7.8.2",
122-
"typescript": "^5.8.3",
122+
"typescript": "^5.9.3",
123123
"wireit": "^0.15.0-pre.2"
124124
},
125125
"dependencies": {
@@ -128,4 +128,4 @@
128128
"zod": "^3.25.76",
129129
"zod-to-json-schema": "^3.25.1"
130130
}
131-
}
131+
}
Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
# Copyright 2026 Google LLC
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
import asyncio
16+
import json
17+
import logging
18+
import os
19+
import shutil
20+
from agent import ContactAgent
21+
from a2ui.core.schema.constants import VERSION_0_8
22+
23+
logging.basicConfig(level=logging.INFO)
24+
25+
26+
def save_messages_with_image_copies(messages: list, image_base_url: str, out_path: str):
27+
"""
28+
Saves the A2UI messages payload to the composer tools directory, copies any
29+
local agent images to the composer's public assets folder, and rewrites the
30+
JSON payload URLs to use the new public Next.js paths.
31+
"""
32+
agent_dir = os.path.basename(os.path.abspath("."))
33+
src_images = "images"
34+
dest_images = f"../../../../tools/composer/public/images/{agent_dir}"
35+
36+
# Copy images if they exist
37+
if os.path.exists(src_images):
38+
os.makedirs(dest_images, exist_ok=True)
39+
for file in os.listdir(src_images):
40+
src_path = os.path.join(src_images, file)
41+
dest_path = os.path.join(dest_images, file)
42+
if os.path.isfile(src_path):
43+
if os.path.exists(dest_path):
44+
print(f"Skipping {file}, already exists in {dest_images}")
45+
continue
46+
shutil.copy2(src_path, dest_images)
47+
print(f"Copied new images from {src_images} to {dest_images}")
48+
49+
# Update JSON payload to use the new public asset paths
50+
json_str = json.dumps(messages, indent=2)
51+
# We replace the local serving url with the Next.js public URL
52+
json_str = json_str.replace(image_base_url, f"/images/{agent_dir}/")
53+
54+
with open(out_path, "w") as f:
55+
f.write(json_str)
56+
print(f"Recorded {len(messages)} A2UI message parts to {out_path}")
57+
58+
59+
async def main():
60+
base_url = "http://localhost:10006"
61+
agent = ContactAgent(base_url=base_url)
62+
query = "Find Alex in Marketing"
63+
session_id = "test_session_2"
64+
65+
print(f"Running agent with query: {query}")
66+
67+
messages = []
68+
69+
async for event in agent.stream(query, session_id, ui_version=VERSION_0_8):
70+
parts = event.get("parts", [])
71+
for p in parts:
72+
if p.root.metadata and p.root.metadata.get("mimeType") == "application/json+a2ui":
73+
# Some payloads are already a list, some are dicts
74+
if isinstance(p.root.data, list):
75+
messages.extend(p.root.data)
76+
else:
77+
messages.append(p.root.data)
78+
79+
if messages:
80+
out_path = "../../../../tools/composer/src/data/dojo/contact-lookup.json"
81+
image_base_url = f"{base_url}/static/"
82+
save_messages_with_image_copies(messages, image_base_url, out_path)
83+
else:
84+
print("No A2UI messages produced.")
85+
86+
87+
if __name__ == "__main__":
88+
asyncio.run(main())
Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
#!/bin/bash
2+
# Copyright 2026 Google LLC
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License");
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
16+
source ~/.openclaw/credentials/secrets.sh
17+
export GEMINI_API_KEY="$GEMINI_API_KEY_ALAN_WORK"
18+
uv run record_scenario.py

0 commit comments

Comments
 (0)