chat-rich-input Track A: AROS attach sheet (photo/file/camera/barcode) + UPC read tool - #193
Open
Nirpat3 wants to merge 3 commits into
Open
chat-rich-input Track A: AROS attach sheet (photo/file/camera/barcode) + UPC read tool#193Nirpat3 wants to merge 3 commits into
Nirpat3 wants to merge 3 commits into
Conversation
…amera, barcode
Adds the shared rich-input affordance to all three AROS chat composers
(Concierge, floating FAB, /start landing) so an owner can SHOW the assistant
an invoice, a price tag, or a product instead of typing it out.
One component, three mounts:
- AttachSheet ("+" -> Photo · File · Camera · Barcode · Voice) is self-contained
(inline styles) so it drops into each composer's own styling system unchanged.
- Attachments send as attachments:[{name,type,dataUrl}] alongside the existing
text path — purely additive, remove it and today's behavior returns.
- Images are client-downscaled (long edge 2000px, JPEG q0.85) before encoding;
10MB/file and 20MB/turn caps are enforced with a graceful human message
instead of an oversized upload crashing the composer.
- Thumbnails render in the composer (removable) and in the transcript; ChatMsg
gains an optional attachments field.
Camera and barcode:
- Camera uses a capture="environment" input, plus getUserMedia for live scan.
- Barcode decodes with the native BarcodeDetector where present and falls back
to a BUNDLED zxing-wasm decoder for iPad Safari / Firefox (the wasm is emitted
as a local asset — no CDN, works offline). A manual UPC field is always
present as the floor, so a denied camera degrades to a working path and never
a dead black pane.
- UPC validation (UPC-A/EAN-13/EAN-8/GTIN-14 mod-10) is pure and unit-tested.
Honest failure, never fabrication:
- A failed attachment turn says "I couldn't read that attachment — I won't guess
what it contains", never a described image the model never saw.
- resolveCatalogState() resolves the three distinct barcode states
(not-connected / not-found / catalog-unreachable) and only reports "found" for
a real named item. A connected-store scan is sent as an explicit
"do not guess" lookup on the real store-data path; with no store connected the
not-connected state is surfaced client-side instead of a sample product.
Style: functional core (attachments.ts — caps, downscale math, UPC checksum,
catalog-state resolution; 17 unit tests) with the browser I/O confined to a thin
shell (encode.ts, decode.ts, and the three components).
Storage seam: transcript persistence strips heavy base64 dataUrls so it cannot
blow the localStorage quota; durable attachment history is left to Shared S
(shre-files + retention) behind that seam.
Voice is NOT built here — the sheet carries a clearly-marked mount slot and a
TODO for the shared voice-everywhere component, and never fakes recording.
Journey spec: docs/journeys/chat-attach-rich-input.md (+ README index).
Gate: vitest 458/458 green (17 new), apps/web tsc --noEmit clean, production
vite build clean with the wasm bundled locally.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ng, scanner, honesty
Review found the feature could not complete its journey on any surface. Fixes:
B1 attachments could never reach the router. The same-origin /v1/chat proxy
capped bodies at 64KB and destroyed the socket, so every real photo/PDF turn
aborted as an opaque "Failed to fetch" and the proxy forwarded the literal
`null` upstream. Raise the chat body limit, drain instead of destroying, and
answer a real 413/400. Client caps are now DERIVED from the transport (nginx
client_max_body_size 10m, base64 x4/3): 6MB/file, 7MB/turn, and the copy quotes
those numbers instead of promising 20MB the edge rejects.
B2 the attach menu rendered offscreen. `position:absolute; bottom:calc(100%+8px)`
on a SIBLING of the trigger with no positioned ancestor on any of the three
composers. Wrap trigger+menu in a relative container (the shell's own
.rsx2-top__menu pattern).
B3 the barcode scanner was a dead black pane. srcObject was assigned while the
<video> was still unmounted, so the stream never attached; the decode loop then
spun at 60fps forever with the camera on. Mount the video unconditionally and
throttle every branch of the loop.
B4 fabrication on 2 of 3 surfaces. ArosChat and StartChat sent scanned UPCs
unguarded — and StartChat sends demoMode:true, so a real scan could return a
plausible SAMPLE product. All three now resolve not-connected honestly; the demo
surface never sends a scan at all.
B5 dead states. resolveCatalogState had no production caller and two of the four
catalog states had no renderer. New CatalogNotice renders every non-found state
with its own CTA, wired via a new pure `barcodeOutcome` resolver.
B6 an attachment turn tripping a text-only intent interceptor was answered from
store data with the file silently discarded. Skip the interceptors when
attachments are present.
B7 caps ran AFTER encoding (a 500MB pick ballooned the tab); no count cap; HEIC
was forwarded verbatim to a provider that rejects it; legacy .doc/.xls/.ppt fell
through to a UTF-8 decode of binary and fed the model mojibake. Cap on file.size
first, cap the count, transcode HEIC to JPEG (honest message when the browser
can't), drop legacy Office.
Shared rails, all three composers:
- draft safety: text AND attachments restored on failure, with retry copy
- Escape + focus management on the menu and the scanner (real menu semantics,
roving arrows, focus trap, restore on close)
- camera-denied is honest and recoverable on both the capture picker (cancel
event) and the scanner ("Try camera again")
- send blocked while files encode, with an aria-live reading state
- remove-by-id instead of a closed-over index (double-tap deleted two files);
44px touch targets; idempotent scanner finish; one timer type in the loop
- role="alert", one error per rejected file, cleared when an attachment is pulled
- muted Voice row now mutes its LABEL and is hidden where a working mic exists
Also: ATTACH_ACCEPT was dead so "File" couldn't pick an image; files with an
empty file.type were wrongly rejected (extension fallback added); a live
FileList was captured before `value=''` cleared it, so every pick silently
vanished; chatHistory serialized full base64 into localStorage (quota bomb).
Adds e2e/chat-attach-rich-input.spec.ts — golden path from the entry point,
asserting the menu is visible AND inside the viewport, a file stages and
removes, an unsupported pick gets a reason, Escape restores focus. It fails
against the old B2 code.
Tests: vitest 480/480, web tsc clean, vite build clean, playwright 14/14 local.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Nirpat3
force-pushed
the
feat/chat-rich-input-aros
branch
3 times, most recently
from
July 23, 2026 08:12
b3fb4ef to
d2e9493
Compare
…s; make the 413 real
Round-2 review found two survivors of the attach-sheet fix.
A1 (blocking) the silent attachment drop was only half fixed. The server-side
gate landed (hasAttachments() fences the /v1/chat intent handlers), but
ConciergeChat kept its own text-only interceptor: EXTERNAL_INTELLIGENCE_REQUEST
matches weather|forecast|temperature|news|headlines|search the web|..., and its
branch returns AFTER the optimistic setDraft('')/setPending([]) and WITHOUT
calling restoreDraft(). So a photographed invoice captioned "any news on this
vendor?" was answered by a canned routing message and the file was discarded
unrecoverably - re-photograph the invoice.
The interceptor now goes through a pure shouldInterceptTextOnly() rail in the
new apps/web/src/redesign/chatIntent.ts: a turn carrying attachments is never
interceptable, mirroring the server's hasAttachments() gate so both ends of the
wire fail the same way. The regex moves into the same pure module so the test
drives the production matcher rather than a copy.
A2 the friendly 413 was unreachable in production and quoted a number nothing
honoured. CHAT_BODY_LIMIT sat at 12MB, ABOVE the nginx client_max_body_size 10m
in front of the process, so the edge always answered first with an opaque HTML
413 and readJsonBody's message was dead code; the message itself advertised
"limit 12 MB", which is neither the transport ceiling (10MB) nor the client cap
(6MB/file, 7MB/turn) a user's file is measured against.
CHAT_BODY_LIMIT is now DERIVED from the caps the client actually enforces -
base64WireBytes(7MB) + 256KB envelope = ~9.58MB, clamped under the 10MB edge -
so every legal turn passes and an over-cap turn gets OUR answer. readJsonBody
takes the over-limit message from its caller, and the chat one quotes the
per-file and per-message limits the user can act on. nginx is left alone (that
is a deploy decision).
Tests: vitest 486/486 (6 new in chatIntent.test.ts, incl. the regression -
an attachment turn whose caption says news/forecast/temperature is NOT
intercepted). apps/web tsc --noEmit clean; root tsc unchanged (66 pre-existing
errors in billing/tasks, none in src/server.ts); vite build clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Track A of the chat-rich-input mission (
shre-dev-kit docs/missions/chat-rich-input.md).What
One shared attach affordance ("+" → Photo · File · Camera · Barcode · Voice) mounted on all three AROS chat composers — Concierge (
ConciergeChat.tsx), floating FAB (ArosChat.tsx), and the/startlanding (StartChat.tsx). One component, three mounts. A store owner can now show the assistant an invoice, a price tag, or a product instead of typing it out.How
attachments:[{name,type,dataUrl}]alongside the existing textfetchto/v1/chat. Purely additive behind the current text path.ChatMsggains an optionalattachmentsfield.capture="environment"input +getUserMedialive scan; permission-denied degrades to Photo/manual, never a dead black pane.BarcodeDetectorwhere present, a bundledzxing-wasmfallback for iPad Safari / Firefox (wasm emitted as a local asset — no CDN, offline-safe), and an always-present manual UPC field. UPC-A/EAN-13/EAN-8/GTIN-14 mod-10 validation is pure + unit-tested.resolveCatalogState()yields the three distinct barcode states (not-connected / not-found / catalog-unreachable) and only reports found for a real named item; a connected-store scan is sent as an explicit "do not guess" lookup on the real store-data path.Style
Functional core (
attach/attachments.ts— caps, downscale math, UPC checksum, catalog-state resolution; 17 unit tests) with browser I/O confined to a thin shell (encode.ts,decode.ts, and the components).Journey
docs/journeys/chat-attach-rich-input.md(+ README index) — golden path, all failure states, honest-failure success signal.Gate
apps/webtsc --noEmitcleanvite buildclean —zxing_reader.wasmbundled locallyThe contract's backend UPC-tool anchors for Track A do not exist anywhere in the
arosrepo (any ref) and were not built here — they describe the shreai monorepo, not this submodule:@shreai/client/buildChatRequest— not a dependency of aros; all three composers use rawfetch+chatReplyText. No migration performed (kept raw fetch, addedattachments).skills/manifests/aros-pos-skill-tools.json— nomanifests/dir exists; AROS skills are data-driven analytics skills inskills/src/skills/*.ts.skills/src/skills/pos-catalog-crud-draft.tsmatchesSelector/selector.upcs— do not exist.The deterministic UPC-keyed catalog read tool belongs in the shreai skills/router repo (Shared Y / Integrator). This PR wires the full client-side barcode workstream and the honest three-state seam so it lights up the moment that backend tool + the Shared Z router pin land.
Scope guard
No deploy, no restart, no merge. Additive and reversible; each piece reverts independently.
🤖 Generated with Claude Code