Skip to content
This repository was archived by the owner on Oct 27, 2026. It is now read-only.

Commit 5bd7083

Browse files
teallarsoncursoragentclaude
authored
Generate toolkit docs from the experience API public catalog (#1183)
* Add public catalog API source for toolkit docs generation. Switch the nightly workflow to unauthenticated /v1/public/* endpoints, introduce PublicCatalogApiSource with toolkit-level auth fan-out, and deprecate the legacy tool-metadata path while renaming source interfaces without the I prefix. Co-authored-by: Cursor <cursoragent@cursor.com> * Strip the arcade- provider prefix from generated toolkit data. The public catalog reports Arcade-hosted OAuth providers as arcade-github, arcade-microsoft-powerbi, and so on. providerId becomes the auth provider docs slug (/en/references/auth-providers/<providerId>), and those pages are named github and microsoft-powerbi, so the prefix would 404 the auth provider link on every toolkit page the next nightly regenerated. Providers that already arrive unprefixed, like salesforce and zendesk and the rest of the BYOC set, match their page and pass through untouched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Remove deprecated toolkit API sources and legacy CLI flags. Delete the authenticated engine and arcade sources, drop I-prefixed interface aliases, and rename the CLI host flag to --api-url with ARCADE_API_URL support while keeping ENGINE_API_URL as a fallback for existing workflow secrets. Co-authored-by: Cursor <cursoragent@cursor.com> * Read toolkit branding from the experience API public catalog. The experience API merges the engine's public catalog with design-system branding, so one anonymous read now supplies both tools and toolkit metadata. PublicCatalogApiSource implements MetadataSource alongside ToolDataSource and serves both from a single cached snapshot, which means tools and branding can no longer describe different states. Provider-name resolution and --require-complete filtering consult the same read, so a toolkit is never dropped for branding the catalog actually has. Only the mock path still builds a separate metadata source, and it does so on demand, so a live run never loads the design-system package. --api-url now points at the experience API and defaults to https://experience.arcade.dev/api. The /v1 prefix injection is gone, since the catalog paths hang off /api. ARCADE_API_URL and ENGINE_API_URL are no longer consulted for the base URL: an engine host builds a path that 404s against this catalog. PUBLIC_CATALOG_URL replaces them, and the nightly needs no URL secret at all. Verified against production: all 123 generated toolkits carry branding and service domains identical to the committed data. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Treat a blank catalog URL as unset. The nightly wires PUBLIC_CATALOG_URL from an optional secret, and GitHub Actions hands an unset secret to the job as "" rather than leaving it absent. Nullish coalescing accepted that empty string as a real base URL, so the run built a relative /public/tool_catalog and died on "Failed to parse URL" instead of falling through to the production default. resolveApiBaseUrl now trims and treats blank as absent, for both the flag and the environment variable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent cbf3250 commit 5bd7083

30 files changed

Lines changed: 1271 additions & 2564 deletions

‎.github/workflows/generate-toolkit-docs.md‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,20 +5,19 @@ This workflow regenerates toolkit JSON and opens a PR with the changes. It can b
55
## What it does
66

77
1. Builds the toolkit docs generator.
8-
2. Generates toolkit JSON in `toolkit-docs-generator/data/toolkits` using the Engine tool metadata and summary endpoints.
8+
2. Generates toolkit JSON in `toolkit-docs-generator/data/toolkits` using the experience API's public tool catalog, which serves tool definitions and toolkit branding in one anonymous read.
99
3. Syncs integrations sidebar navigation from the generated JSON.
1010
4. Creates or updates a PR on the stable `automation/toolkit-docs` branch if any files changed. Later runs overwrite that open PR with the latest generated docs.
1111

1212
## Inputs and secrets
1313

1414
Required secrets:
1515

16-
- `ENGINE_API_URL`
17-
- `ENGINE_API_KEY`
1816
- `ANTHROPIC_API_KEY`
1917

2018
Optional secrets:
2119

20+
- `PUBLIC_CATALOG_URL` (falls back to `https://experience.arcade.dev/api`; set it to run against staging)
2221
- `ANTHROPIC_MODEL` (falls back to `claude-sonnet-4-6`)
2322
- `ANTHROPIC_EDITOR_MODEL` (falls back to `claude-sonnet-4-6`)
2423

‎.github/workflows/generate-toolkit-docs.yml‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
name: Generate toolkit docs
2-
# Description: Generate toolkit JSON from Engine API, then sync sidebar navigation.
2+
# Description: Generate toolkit JSON from the public tool catalog, then sync sidebar navigation.
33
# Flow:
44
# 1) Generate toolkit JSON into toolkit-docs-generator/data/toolkits
55
# 2) Sync integrations sidebar _meta.tsx from toolkit-docs-generator/data/toolkits
@@ -65,9 +65,7 @@ jobs:
6565
--skip-unchanged \
6666
--preserve-last-known-good \
6767
--verbose \
68-
--api-source tool-metadata \
69-
--tool-metadata-url "$ENGINE_API_URL" \
70-
--tool-metadata-key "$ENGINE_API_KEY" \
68+
--api-source public-catalog \
7169
--llm-provider anthropic \
7270
--llm-model "$ANTHROPIC_MODEL" \
7371
--llm-api-key "$ANTHROPIC_API_KEY" \
@@ -85,8 +83,10 @@ jobs:
8583
echo "succeeded=true" >>"$GITHUB_OUTPUT"
8684
working-directory: toolkit-docs-generator
8785
env:
88-
ENGINE_API_URL: ${{ secrets.ENGINE_API_URL }}
89-
ENGINE_API_KEY: ${{ secrets.ENGINE_API_KEY }}
86+
# Unset in normal runs: the generator defaults to the production
87+
# experience API, which needs no credentials. Set the secret to
88+
# point a run at staging or a local BFF.
89+
PUBLIC_CATALOG_URL: ${{ secrets.PUBLIC_CATALOG_URL }}
9090
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
9191
ANTHROPIC_MODEL: ${{ secrets.ANTHROPIC_MODEL || 'claude-sonnet-4-6' }}
9292
# Stronger model for the secret-coherence editor. Keeps

‎toolkit-docs-generator/ARCHITECTURE.md‎

Lines changed: 16 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -19,14 +19,13 @@ flowchart TD
1919
manual["Manual run<br/>workflow_dispatch"] --> generate
2020
porter["Porter deploy succeeded<br/>repository_dispatch"] --> generate
2121
22-
engine["Engine API<br/>/v1/tool_metadata"] -->|"tools, parameters, auth, secrets"| generate
22+
catalog["Public tool catalog (experience API)<br/>public/tool_catalog + public/tools"] -->|"tools, parameters, auth, secrets,<br/>category, label, icon, docsLink, flags"| generate
2323
previous["data/toolkits/*.json<br/>previous run"] -->|"signatures and curation hashes"| generate
2424
2525
generate["generate --all --skip-unchanged"] --> changed{"Changed since<br/>last run?"}
2626
changed -->|no| skipped["Left untouched"]
2727
changed -->|yes| merger
2828
29-
design["@arcadeai/design-system<br/>TOOLKITS"] -->|"category, label, icon, docsLink, flags"| merger
3029
author["Docs author"] -->|"writes and reviews in a PR"| curation
3130
curation["curation/toolkit/<br/>chunks/*.mdx<br/>imports/*.mdx<br/>pages/**/*.mdx"] -->|"documentationChunks, customImports, subPages"| merger
3231
anthropic["Anthropic"] -->|"code examples, summaries, secret edits"| merger
@@ -47,11 +46,18 @@ flowchart TD
4746
```
4847

4948
Each field in a toolkit JSON file comes from exactly one of those inputs. The
50-
Engine API owns everything about a tool, the design system owns everything about
51-
a toolkit's placement and identity, `curation/` owns hand-authored prose, and the
52-
LLM owns code examples and summaries. When a source fails, the merger falls back
53-
to the previous run's file rather than inventing a value, and reports what it
54-
recovered.
49+
public catalog owns everything about a tool and about a toolkit's placement and
50+
identity, `curation/` owns hand-authored prose, and the LLM owns code examples
51+
and summaries. When a source fails, the merger falls back to the previous run's
52+
file rather than inventing a value, and reports what it recovered.
53+
54+
The catalog is one read for both halves. The experience API joins the engine's
55+
operational catalog to design-system branding before the generator sees it, so
56+
tools and branding always describe the same snapshot. The generator no longer
57+
reads `@arcadeai/design-system` itself except through the deprecated engine
58+
sources. When the experience API's branding read fails it returns
59+
`metadata: null` for a toolkit; the merger substitutes defaults and flags it,
60+
and `--require-complete` fails the run instead.
5561

5662
`curation/` is the one input a person edits directly. Those Markdown and MDX
5763
files live in this repository and merge like any other change, so a prose edit
@@ -66,9 +72,8 @@ it. The sidebar sync writes navigation only, and never touches toolkit JSON.
6672

6773
### Data sources
6874

69-
- `EngineApiSource` fetches tool metadata from the Engine API.
70-
- `ArcadeApiSource` fetches tool metadata from the Arcade API.
71-
- `DesignSystemMetadataSource` loads toolkit metadata from `@arcadeai/design-system`.
75+
- `PublicCatalogApiSource` fetches tools and toolkit branding from the public
76+
tool catalog on the experience API, serving both from one cached snapshot.
7277
- `MarkdownCurationSource` compiles documentation chunks, import declarations,
7378
and subpages from the configured curation directory. When configured, that
7479
directory is globally authoritative: a missing toolkit directory means the
@@ -138,7 +143,7 @@ public, read-only values configured through these Vercel environment variables:
138143

139144
## Key files
140145

141-
- `src/sources/engine-api.ts` — tool metadata from Engine API
146+
- `src/sources/public-catalog-api.ts` — tools and toolkit branding from the public catalog
142147
- `src/sources/markdown-curation.ts` — Markdown and MDX curation compiler
143148
([format reference](CURATION.md))
144149
- `src/sources/toolkit-data-source.ts` — unified data source

‎toolkit-docs-generator/CURATION.md‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -251,8 +251,7 @@ generation PR, or generate that one toolkit with Engine credentials and run
251251
```bash
252252
../node_modules/.bin/tsx src/cli/index.ts generate \
253253
--providers "GoogleFlights" \
254-
--tool-metadata-url "$ENGINE_API_URL" \
255-
--tool-metadata-key "$ENGINE_API_KEY" \
254+
--api-url "$ARCADE_API_URL" \
256255
--custom-sections ./curation \
257256
--skip-examples --skip-summary --skip-secret-coherence \
258257
--output data/toolkits

‎toolkit-docs-generator/README.md‎

Lines changed: 31 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -11,36 +11,45 @@ This guide explains how Arcade generates toolkit JSON, how the GitHub workflow r
1111

1212
## Data sources
1313

14-
The generator merges three inputs into one JSON output per toolkit:
14+
The generator merges two inputs into one JSON output per toolkit:
1515

16-
- **Engine API** for tool definitions, auth requirements, and scopes
17-
- **Arcade design system** for metadata (category, icons, flags)
16+
- **Public tool catalog** on the experience API for tool definitions, auth
17+
requirements, scopes, and toolkit branding (category, label, icon, docs link,
18+
pro/BYOC flags) in a single anonymous read
1819
- **Custom sections file** (optional) for preserved or curated docs content
1920

2021
It also reads the previous output when you use `--skip-unchanged` or `--previous-output`.
2122

22-
When `--skip-unchanged` runs against the tool metadata API, the generator fetches
23-
one complete snapshot from `/v1/tool_metadata`. It reuses that snapshot for
24-
change detection, progress calculation, and generation so a run cannot compare
25-
different API states. Only changed toolkits are regenerated.
23+
The catalog lives at `https://experience.arcade.dev/api` by default; override it
24+
with `--api-url` or `PUBLIC_CATALOG_URL`. The generator fetches one
25+
complete snapshot from `public/tool_catalog` and `public/tools`, then reuses it
26+
for change detection, progress calculation, and generation so a run cannot
27+
compare different API states. Only changed toolkits are regenerated.
28+
29+
The design-system package (`@arcadeai/design-system`) is only consulted by the
30+
deprecated engine sources. When the experience API cannot reach the design
31+
system it returns `metadata: null` for a toolkit, and the merger falls back to
32+
default branding and flags the toolkit; `--require-complete` turns that into a
33+
hard failure instead.
2634

2735
## GitHub workflow: generate and sync
2836

2937
The workflow file is `/.github/workflows/generate-toolkit-docs.yml`.
3038
It runs these steps:
3139

3240
1. Type-check and test the toolkit docs generator.
33-
2. Generate toolkit JSON using `toolkit-docs-generator` and the Engine API.
41+
2. Generate toolkit JSON using `toolkit-docs-generator` and the public tool catalog.
3442
3. Sync sidebar navigation from `toolkit-docs-generator/data/toolkits` to the `_meta.tsx` files.
3543
4. Create or update a pull request if there are changes.
3644

3745
Required secrets:
3846

39-
- `ENGINE_API_URL`, `ENGINE_API_KEY`
4047
- `ANTHROPIC_API_KEY` for examples, summaries, and secret-coherence edits
4148

4249
Optional secrets:
4350

51+
- `PUBLIC_CATALOG_URL` (defaults to `https://experience.arcade.dev/api`; set it
52+
to point a run at staging or a local BFF)
4453
- `ANTHROPIC_MODEL` (defaults to `claude-sonnet-4-6` in the workflow)
4554
- `ANTHROPIC_EDITOR_MODEL` (defaults to `claude-sonnet-4-6` in the workflow)
4655

@@ -107,8 +116,7 @@ The summary generator is configured to **never list OAuth scopes** in the genera
107116
```bash
108117
pnpm dlx tsx src/cli/index.ts generate \
109118
--providers "Github" \
110-
--tool-metadata-url "$ENGINE_API_URL" \
111-
--tool-metadata-key "$ENGINE_API_KEY" \
119+
--api-source public-catalog \
112120
--llm-provider openai \
113121
--llm-model gpt-4.1-mini \
114122
--llm-api-key "$OPENAI_API_KEY" \
@@ -121,7 +129,7 @@ pnpm dlx tsx src/cli/index.ts generate \
121129
## Architecture at a glance
122130

123131
- **CLI**: `toolkit-docs-generator/src/cli/index.ts`
124-
- **Sources**: `src/sources` for Engine API, mock data, and design system metadata
132+
- **Sources**: `src/sources` for the public catalog, mock data, and the deprecated engine APIs
125133
- **Merger**: `src/merger/data-merger.ts` merges tools, metadata, and custom sections
126134
- **Generator**: `src/generator/json-generator.ts` writes JSON and `index.json`
127135
- **Verifier**: `src/generator/output-verifier.ts` validates output and index consistency
@@ -133,13 +141,15 @@ Run these commands from the `toolkit-docs-generator` directory. Invoke `tsx` by
133141
path because this directory has no package for `pnpm exec`; the sidebar sync
134142
command below runs from the repo root instead.
135143

144+
`--api-source public-catalog` is spelled out below because a bare run with no
145+
catalog or engine environment variables falls back to the mock fixtures.
146+
136147
Generate a single toolkit:
137148

138149
```bash
139150
pnpm dlx tsx src/cli/index.ts generate \
140151
--providers "Github:1.0.0" \
141-
--tool-metadata-url "$ENGINE_API_URL" \
142-
--tool-metadata-key "$ENGINE_API_KEY" \
152+
--api-source public-catalog \
143153
--llm-provider openai \
144154
--llm-model gpt-4.1-mini \
145155
--llm-api-key "$OPENAI_API_KEY" \
@@ -152,8 +162,7 @@ Generate all toolkits:
152162
pnpm dlx tsx src/cli/index.ts generate \
153163
--all \
154164
--skip-unchanged \
155-
--tool-metadata-url "$ENGINE_API_URL" \
156-
--tool-metadata-key "$ENGINE_API_KEY" \
165+
--api-source public-catalog \
157166
--llm-provider openai \
158167
--llm-model gpt-4.1-mini \
159168
--llm-api-key "$OPENAI_API_KEY" \
@@ -165,8 +174,7 @@ Generate without LLM output:
165174
```bash
166175
pnpm dlx tsx src/cli/index.ts generate \
167176
--providers "Asana:0.1.3" \
168-
--tool-metadata-url "$ENGINE_API_URL" \
169-
--tool-metadata-key "$ENGINE_API_KEY" \
177+
--api-source public-catalog \
170178
--skip-examples \
171179
--skip-summary \
172180
--output data/toolkits
@@ -231,8 +239,10 @@ deletes it and rebuilds `index.json`.
231239
- `--all` generate all toolkits
232240
- `--providers` generate a subset of toolkits
233241
- `--skip-unchanged` only write changed toolkits
234-
- `--api-source` select `tool-metadata` (default with Engine creds), `list-tools`
235-
(only with the explicit flag), or `mock`
242+
- `--api-source` select `public-catalog` or `mock`. Auto-detects
243+
`public-catalog` when `PUBLIC_CATALOG_URL` is set, and `mock` otherwise
244+
- `--api-url` public catalog base URL including path prefix
245+
(default `https://experience.arcade.dev/api`, or `PUBLIC_CATALOG_URL`)
236246
- `--previous-output` compare against a previous output directory
237247
- `--custom-sections` load an authoritative Markdown/MDX curation directory
238248
- `--skip-examples`, `--skip-summary` disable LLM steps
@@ -298,5 +308,5 @@ message and the third, because those are two separate things to fix.
298308
## Troubleshooting
299309

300310
- **Nothing regenerated**: `--skip-unchanged` exits early when tool definitions did not change.
301-
- **Missing metadata**: the generator falls back to the metadata JSON file when design system metadata is unavailable.
311+
- **Missing metadata**: when the catalog returns `metadata: null` for a toolkit the merger substitutes defaults and warns. Re-run once the experience API's design-system read recovers, or use `--require-complete` to fail instead of publishing defaults.
302312
- **Verify output fails**: run `pnpm dlx tsx src/cli/index.ts verify-output --output data/toolkits` and fix the reported mismatch.
Lines changed: 63 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -1,43 +1,75 @@
1-
export type ApiSource = "list-tools" | "tool-metadata" | "mock";
1+
export type ApiSource = "public-catalog" | "mock";
22

33
type ApiSourceOptions = {
44
apiSource?: string;
5-
toolMetadataUrl?: string;
6-
toolMetadataKey?: string;
5+
apiUrl?: string;
76
};
87

9-
export const resolveApiSource = (options: ApiSourceOptions): ApiSource => {
10-
// Explicit source takes precedence
11-
if (options.apiSource) {
12-
const source = options.apiSource.toLowerCase();
13-
if (source === "list-tools") {
14-
return "list-tools";
15-
}
16-
if (source === "engine") {
17-
return "tool-metadata";
18-
}
19-
if (source === "tool-metadata") {
20-
return "tool-metadata";
21-
}
22-
if (source === "mock") {
23-
return "mock";
24-
}
25-
throw new Error(
26-
`Invalid --api-source "${options.apiSource}". Use "list-tools", "tool-metadata", or "mock".`
27-
);
8+
const EXPLICIT_API_SOURCES: Record<string, ApiSource> = {
9+
"public-catalog": "public-catalog",
10+
public: "public-catalog",
11+
mock: "mock",
12+
};
13+
14+
const resolveExplicitApiSource = (apiSource: string): ApiSource => {
15+
const resolved = EXPLICIT_API_SOURCES[apiSource.toLowerCase()];
16+
if (resolved) {
17+
return resolved;
2818
}
2919

30-
// Auto-detect based on provided Engine credentials only.
31-
// List-tools endpoint must be explicitly selected via --api-source list-tools.
32-
const hasToolMetadataKey = !!(
33-
options.toolMetadataKey ?? process.env.ENGINE_API_KEY
34-
);
35-
const hasToolMetadataUrl = !!(
36-
options.toolMetadataUrl ?? process.env.ENGINE_API_URL
20+
throw new Error(
21+
`Invalid --api-source "${apiSource}". Use "public-catalog" or "mock".`
3722
);
23+
};
24+
25+
/**
26+
* Pick a source from the environment when --api-source is omitted.
27+
*
28+
* An explicitly configured URL means the caller wants live data. With nothing
29+
* set, stay on fixtures rather than reaching out to the network behind the
30+
* caller's back, even though the public catalog needs no credentials and
31+
* carries its own default URL.
32+
*/
33+
const resolveAutoDetectedApiSource = (options: ApiSourceOptions): ApiSource => {
34+
const hasApiUrl = resolveApiBaseUrl(options.apiUrl) !== undefined;
3835

39-
if (hasToolMetadataKey && hasToolMetadataUrl) {
40-
return "tool-metadata";
36+
if (hasApiUrl) {
37+
return "public-catalog";
4138
}
39+
4240
return "mock";
4341
};
42+
43+
/**
44+
* Treat a blank value as absent.
45+
*
46+
* An unset GitHub Actions secret still reaches the job as `""`, and `??` would
47+
* accept that as a real base URL and build a relative `/public/tool_catalog`
48+
* that fetch cannot parse. The nightly wires PUBLIC_CATALOG_URL from an
49+
* optional secret, so this is the normal case, not an edge case.
50+
*/
51+
const blankToUndefined = (value: string | undefined): string | undefined => {
52+
const trimmed = value?.trim();
53+
return trimmed ? trimmed : undefined;
54+
};
55+
56+
/**
57+
* The catalog now lives on the experience API, whose paths hang off `/api`
58+
* rather than the engine's `/v1`. An engine host would build a URL that 404s,
59+
* so ARCADE_API_URL and ENGINE_API_URL are deliberately not consulted here.
60+
*/
61+
export const resolveApiBaseUrlFromEnv = (): string | undefined =>
62+
blankToUndefined(process.env.PUBLIC_CATALOG_URL);
63+
64+
/** The configured base URL, or undefined when neither flag nor env supplies one. */
65+
export const resolveApiBaseUrl = (
66+
apiUrl: string | undefined
67+
): string | undefined => blankToUndefined(apiUrl) ?? resolveApiBaseUrlFromEnv();
68+
69+
export const resolveApiSource = (options: ApiSourceOptions): ApiSource => {
70+
if (options.apiSource) {
71+
return resolveExplicitApiSource(options.apiSource);
72+
}
73+
74+
return resolveAutoDetectedApiSource(options);
75+
};

0 commit comments

Comments
 (0)