Repository navigation
Promote Payload cutover to production - #113
Merged
Merged
Conversation
Adds /tools/ai-chatbot: a visitor pastes a URL, we crawl up to 20 pages, they chat with a bot built from that corpus, and we show a scored report on what their site cannot answer. Verified email holders can then claim an embed key and run the same bot on their own site. Crawler (lib/crawl/, shared with the planned Website Grader and llms.txt Generator): - URL safety gate; every redirect hop re-validated, not just the seed - robots.txt parsing, sitemap discovery via the Sitemap: directive, and one level of sitemap index (most real sitemaps are indexes) - HTML to text with heading metadata in a single pass - caps: 20 pages, 500KB/page, 20s wall clock, 5 concurrent, 3 redirects, identified User-Agent Platform (lib/tools/): - atomic daily counters, corrected from the spec's UPDATE form which reported "capped" on the first request of every day - OpenRouter client with a model chain; reasoning disabled after it leaked chain-of-thought into answers and made replies 7x slower - site-checks.ts: structure, crawlability and specificity, tool-agnostic Chatbot (lib/tools/chatbot/): - system prompt that answers only from the crawled text and says so plainly when it cannot - report: three measured scores at crawl time, answerability and coverage from one background model call, evidence required per question - embed keys bound to the crawled host, per-key daily cap, attribution - email verification: HMAC'd codes, 10 min expiry, 5 attempts, single use, 3 per email per day, 3 per session. Sending is stubbed until a provider exists; codes are logged Also: CI ran on branch names this repo no longer uses, so it never ran on a pull request. Fixed, moved to Node 22, and given a test step. 75 tests. Known gaps are tracked in docs/ai-chatbot-plan.md. The route layer has no automated tests, nothing prunes stored page text yet, and the privacy line is unwritten. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Secrets needed, migrations, where the verification code appears while email sending is stubbed, and how to preview the embed widget. Also records that OpenRouter's free quota is account-wide and needs $10 of credits to be usable, since that is what stops the tool working first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ecrawl Origin binding no longer strips labels. Deriving an apex from a hostname needs the Public Suffix List, and without it `random.vercel.app` reduces to `vercel.app`, which would let one key answer for every site Vercel hosts. The same holds for `*.myshopify.com`, `*.github.io` and `*.framer.website`. The bound host is now stored as crawled and compared as stored: exact host, `www`, and localhost. The subdomain wildcard is gone for the same reason. Localhost is accepted for any key, so a customer's developer can test before installing, and those messages are counted separately (`embed-dev:<key>`, 20 a day) from live traffic (`embed:<bound_host>`, 50 a day). Two consequences: a laptop can never spend a live site's allowance, and a domain that crawls three times and claims three keys still gets 50 messages rather than 150. This replaces `TOOLS_EMBED_ALLOW_ANY_ORIGIN`, which was a global switch we could never hand to a customer. A rejected origin now names the bound host, because "not available" reads as broken to whoever is installing it. No per-IP cap on embed messages. One IP sending 50 messages is either the owner testing after install or someone draining the quota, and there is no way to tell them apart. Capping it punishes the first; the second costs nothing on a free model. Manual recrawl ships as a signed link rather than a button. The widget is shown to the customer's own visitors, and a recrawl points twenty requests at their server, so it cannot be something a stranger can press. The token is an HMAC of the key, so there is no table, no expiry to sweep, and revoking the embed kills the link. Three a day per domain. The embed's session id never changes, so the snippet already pasted into their HTML keeps working — only the pages underneath are swapped, and a recrawl that comes back empty is rejected rather than blanking a live bot. The link is shown on screen at claim time, not emailed: there is still no email provider configured, so a link we only sent by email would reach nobody. Also: widget history moves to sessionStorage, keyed per embed. It was held in a plain variable, so the conversation reset every time a visitor clicked a link. 11 tests on the origin check, mostly the hosting-suffix cases. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ider On a deployed environment the verification code only reaches a Worker log, so nobody outside the team can finish the embed claim — staging exists to be tested, and this was the one flow that could not be. `TOOLS_REVEAL_CODES=true` returns the code in the response, and the client prints it to the browser console rather than the page: it is a testing affordance, not something to show a visitor. Two conditions guard it. The flag must be set, and the send must have actually been stubbed — so configuring RESEND_API_KEY closes it off even if the flag is left behind. It cannot silently follow the code into production, where it would make email verification meaningless. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… redirects Three findings from review, all confirmed against the code. **The salt fallback made manage links forgeable.** Four call sites read `env.TOOLS_IP_SALT ?? 'dev-salt-not-for-production'`. That constant is in the repo, and embed keys are public by design — they sit in the customer's page source — so anyone could compute `manageToken(key, fallback)` and recrawl a customer's site on demand. The same fallback would have made every stored `ip_hash` a plain hash of an address space small enough to enumerate, which is the opposite of what the privacy line claims. There is now one `toolsSalt()` helper returning null when unset, and all four routes return 503 rather than running on a known secret. **The per-session cap was not atomic.** `messages_used` was read at the top of the message route and incremented after the answer, so eight requests fired at once for one session all saw zero used, all called the model, and all recorded a turn. Replaced with the same conditional-update shape the daily counters already use: `reserveTurn` claims one before any model call and returns false at the cap. `releaseTurn` hands it back when the model never answered, so an outage does not cost the visitor a question. `recordTurn` becomes `saveTranscript` and no longer touches the count. **Redirects could outlast the crawl deadline.** `fetchText` re-armed the full timeout on every hop, so one page with three slow redirects could run 4 × budgetMs — up to 32s against a 20s wall clock the route promises. The budget is now an absolute deadline computed once, with each hop getting what is left. 7 tests on the reservation, including the parallel case. Note that node:sqlite executes synchronously, so that test pins the conditional-UPDATE semantics rather than reproducing true concurrency — which is the part that was wrong. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
feat(tools): AI chatbot diagnostic + embeddable widget
Routes, metadata, sitemap and the markdown builders now read through a CMS-neutral provider interface instead of importing Strapi shapes directly, with StrapiProvider and PayloadProvider behind it. Provider selection stays an environment setting so the rollback is a variable, not a revert. Draft reads require CMS_MODE=draft and DEPLOYMENT_ROLE=preview together, and anything else falls back to published, so a single mistyped variable cannot expose drafts on the public domain. A preview deployment additionally serves noindex, no-store, and an empty sitemap. /api/cms/revalidate accepts signed webhooks from Payload, mapping only known collections onto application-owned paths and purging the provider's fetch tags alongside them. Draft saves are ignored on production deployments. scripts/cms-parity.ts compares what the website would render from each provider, on normalised output rather than raw API responses, so a reported difference is one a visitor could see. Differences the migration intends are declared and reported separately from defects: media URLs moving from Strapi derivatives at bare paths to originals at absolute URLs (Cloudflare resizing verified live on both forms), folded duplicate type spellings, and the corrected e-commerce slug. Workflow and wrangler config carry the new variables per environment. Staging becomes the preview deployment; both environments keep CMS_PROVIDER=strapi until cutover. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The file was renamed from .mts to .ts so Node loads the imported providers as CommonJS, but the package script kept the old name. Typecheck passed because tsc does not read package.json scripts, so the break only surfaced on running it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The parity check found that PayloadProvider passed industry and platform through
as bare slugs, where Strapi returned `{ title, slug }` and the normalisers read
`.slug`. Every project therefore read as industry "other" and platform
"website", which silently breaks filtering on /projects — 86 of 90 projects
affected.
A blank alt is now treated as absent so the consumers' `??` fallback to the
document title actually fires; an empty string satisfied it and suppressed the
alt entirely.
Blog and project sorts gain legacyStrapiId as a tiebreaker, matching how Strapi
ordered documents sharing a date.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The CMS moves its SEO fields into plugin-seo's `meta` group. The provider flattens them back to metaTitle and metaDescription so pages, metadata and the parity comparison are unaffected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Current state, the remaining steps with their acceptance checks, and the traps that have already cost time: the migration propagation race, the shared media bucket, generateSlug regenerating migrated slugs, omitted PATCH keys failing to clear values, and the pnpm version constraints. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CMS provider abstraction and Payload provider
Switch staging to Payload CMS
Switch production to Payload CMS
|
Warning XHawk review did not run: out of credits I stopped before reviewing this PR because your workspace has no credits left, so nothing was posted. Add credits at app.xhawk.ai/billing, then start the review again from XHawk and I will pick up the latest commit. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Production release
Required gates completed