Correct pre-built policy pack coverage, and enforce the allowlist against the registry - #21055
Conversation
The pre-built packs table and the pages that point at it were written when ISO 27001, NIST, and PCI DSS shipped for fewer clouds than they do today, and were never refreshed as coverage grew. The generated reference pages under /docs/reference/pre-built-policy-packs/ (driven by data/policy_packs.yaml) have been correct all along, so the summary surfaces were the only thing telling readers we ship less than we do. Reconciled against data/policy_packs.yaml and data/policy_pack_policies/: - ISO/IEC 27001:2022 — listed as AWS-only; we ship AWS, Azure, and Google Cloud (238 + 158 + 153 = 549 policies). - PCI DSS v4.0.1 — listed as AWS-only; we ship all three clouds. - NIST SP 800-53 — listed as AWS and Google Cloud; we also ship Azure. - CMMC 2.0 (AWS) — missing from the table entirely. - HITRUST CSF — missing its AWS Native (aws-native provider) pack. Also added ISO 27001 and CMMC to the framework lists on the Policies overview, the policy packs overview, the Insights landing page, and the onboarding guide, and filled in the missing packs on the per-cloud AWS, Azure, and Google Cloud integration pages. Every link added was checked against the allowlist that generates those pages. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BvCEyksviZ5M9vNaZvAsdz
|
Your site preview for commit 2c284cd is ready! 🎉 http://www-testing-pulumi-docs-origin-pr-21055-2c284cd5.s3-website.us-west-2.amazonaws.com Changed pages: |
github-actions
Bot
added
review:triaging
![unblocked[bot]](https://avatars.githubusercontent.com/in/166219?s=60&v=4){kind=small}
There was a problem hiding this comment.
✅ No issues found
About Unblocked
Unblocked has been set up to automatically review your team's pull requests to identify genuine bugs and issues.
📖 Documentation — Learn more in our docs.
💬 Ask questions — Mention @unblocked to request a review or summary, or ask follow-up questions.
👍 Give feedback — React to comments with 👍 or 👎 to help us improve.
⚙️ Customize — Adjust settings in your preferences.
github-actions
Bot
added
the
review:in-progress
Pre-merge Review — Last updated 2026-08-26T19:22:25ZTip Summary: Nothing outstanding, and nothing left in Two things worth saying about the fix beyond "it landed." First, the relocated paragraph was also reworded from "the same failure this PR was opened to fix" to "the same failure this check exists to catch" — an unprompted improvement, and the right one: a source comment that dates itself to a pull request is the next generation of the drift this change exists to stop. Second, you're right that my suggested header was wrong, and it contradicted my own arithmetic. It said "all 13 come back I re-ran the reconciliation against the file as it now stands: the "must NOT be documented" block names exactly 12 packs, plus Review confidence:
Investigation log
🔍 Verification trail39 claims extracted · 33 verified · 1 unverifiable · 4 contradicted · 1 detector findings
🚨 Outstanding in this PRNo outstanding findings.
|
github-actions
Bot
added
review:no-blockers
| - [NIST 800-53 for AWS](/docs/reference/pre-built-policy-packs/nist/aws/) | ||
| - [PCI DSS for AWS](/docs/reference/pre-built-policy-packs/pci-dss/aws/) | ||
| - [HITRUST CSF for AWS](/docs/reference/pre-built-policy-packs/hitrust/aws/) | ||
| - [HITRUST CSF for AWS Native](/docs/reference/pre-built-policy-packs/hitrust/aws-native/) — for the AWS Cloud Control (`aws-native`) provider. |
There was a problem hiding this comment.
This isn't actually published. It exists in our repo privately. I noticed it's appearing in references, too. Would you mind removing that in this PR?
I think some docs automation might have picked it up @jkodroff
There was a problem hiding this comment.
Good catch (sorry for the Claude-speak). I'll hit you up on Slack to figure out the source of truth on what should actually be published.
There was a problem hiding this comment.
Done in 93d5904 — removed from the table, this page, and data/policy_packs.yaml, which is what was generating the reference page. Also dropped the fetched policy data and its lastmod entry, and added an S3 redirect from /docs/reference/pre-built-policy-packs/hitrust/aws-native/ to the HITRUST AWS page, since the old URL is live today.
On how it got there, since it's relevant to the source-of-truth question: data/policy_packs.yaml is a hand-maintained allowlist, and its rule read "only document packs that are built from the policy-packs-internal repo, i.e. the ones we ship to customers" — treating those two as one condition. hitrust-awsnative satisfies the first and not the second, and nothing available to the docs side distinguishes them: it's built there, it published cleanly in the 2026-07-09 batch, and the Pulumi Cloud API returns it like any other pack, which is how the nightly fetch picked up 114 policies for it. So the automation wasn't working from a bad signal — it was working from the only signal it has, which doesn't carry the private/public bit at all.
I've reworded the rule as two independent conditions and recorded the exclusion at the bottom of that file, so an audit that diffs the org listing against the repo doesn't re-add it. But that's a comment telling a human to go ask, not a check — if there's a durable way to mark a pack private in policy-packs-internal (a private: true in its package.json, a naming convention, anything the fetch script could read), that'd be a much better fix than trusting the next person to read a comment. Happy to wire that up on the docs side once you and @jkodroff land on what the source of truth should be.
Worth a look while you're in there: the same batch had four packs fail to publish and never get retried, one of them aws-organizations-tag-policies v2.0.1 — pulumi/policy-packs-internal#211. Found by an agent, unverified by a human, so treat it accordingly.
Generated by Claude Code
Upstream restructured content/docs/administration/, which moved setting-up-for-success.md from onboarding-guide/ to get-started/. Git followed the rename and carried this branch's edit with it; no conflicts to resolve by hand. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BvCEyksviZ5M9vNaZvAsdz
@danbiwer flagged in review that the HITRUST AWS Native pack is private to policy-packs-internal and is not published to customers, and that docs automation had generated a public reference page for it at /docs/reference/pre-built-policy-packs/hitrust/aws-native/. Removes it from the pre-built packs table, the AWS integrations page, and data/policy_packs.yaml, which is what generated the reference page. Also drops the fetched policy data and its lastmod entry so the data set stays consistent with the allowlist. The reference page is live today, so removing it changes a public URL: adds an S3 redirect to the HITRUST AWS page, the nearest real equivalent. The allowlist rule is what let this through. It said to document packs "built from the policy-packs-internal repo, i.e. the ones we ship to customers", treating those as the same set. They aren't: this pack is built there, publishes cleanly, and is returned by the API, and none of that makes it customer-facing. Restates the rule as two independent conditions, notes that neither the repo nor the API can distinguish them, and records the exclusion at the bottom of the file so a future audit against the org doesn't re-add it. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BvCEyksviZ5M9vNaZvAsdz
github-actions
Bot
added
review:stale
|
@claude #update-review Two pushes since the last review:
The removal spans the table, the AWS integrations page, Not disputing anything else. The out-of-scope note about Generated by Claude Code |
github-actions
Bot
added
review:in-progress
|
🤖 Review updated on @CamSoper's request. |
github-actions
Bot
added
review:no-blockers
| # guide, not a policy listing. Stays hand-maintained. | ||
| # cis-kubernetes-google-cloud A stale earlier publish, superseded by | ||
| # cis-kubernetes-gcp. See the cis-kubernetes section. | ||
| # hitrust-awsnative Built here and resolvable through the API, but NOT a |
There was a problem hiding this comment.
It's resolvable through the API b/c someone published it specifically to the pulumi org. We either need an allowlist on the docs end, or a "clean" org for reading things from API endpoints.
There was a problem hiding this comment.
Built the allowlist option, in cd00d9a2, using the source field you and @danbiwer landed on in Slack.
data/policy_packs.yaml was already an allowlist — it's how we document 22 of the packs that repo carries rather than all of them. What it didn't have was any check on its own entries, so a human adding a wrong row was enough. fetch-policy-packs.js now requires every entry to come back from /api/registry/policypacks?orgLogin=<org> with source: "pulumi", and fails naming the offenders and their actual source.
Verified against the two records Dan pasted, including both failure modes from this incident — a source: "private" pack, and a pack missing from the listing entirely, which is hitrust-awsnative's state now that you've deleted it.
Two things from your thread are captured as comments in the code, because both are easy to get wrong later:
- Not
publisher. It's"pulumi"for everything in our org, so filtering on it admits the private packs too. It's the more obvious-looking field, so there's a comment saying don't switch to it. - Your spoofing question is what made a hard failure defensible. Dan's answer —
sourceis admin-set, not set by whoever publishes — is why the check throws rather than warns. If a passing engineer could set it, gating the build on it would be theater. That reasoning is in the comment too, so it survives.
The check fails closed: an unreadable or changed response stops the job rather than publishing unverified pages.
I couldn't exercise the live call — both endpoints need a token this environment doesn't have — so the parsing is written against Dan's records and unit-tested, and the envelope key is accepted loosely since only the record shape was confirmed. First real run will settle it, and it's on the nightly, so that's tonight.
Which brings up something time-sensitive: deleting the pack from the org means master is now broken for that job. fetch-policy-packs.js throws when an allowlisted pack isn't in the org, and master still carries the entry — so the 07:30 UTC run fails and pings #docs-ops. This PR is the fix.
One more, tangential: Dan's aws-organizations-tag-policies record reads version: 1.0.0, while the repo is at 2.0.1 and there's a GitHub release for 2.0.0. If that was a live listing rather than an illustrative paste, the published version is lagging further than pulumi/policy-packs-internal#211 estimates and I should correct that issue. If it was just a paste, ignore me.
Generated by Claude Code
There was a problem hiding this comment.
All you have to do is look for
source:pulumi. Don't need a static allowlist.
Agreed on the gate, and e7794493 makes that true in both directions. One wrinkle on deleting the file outright, though.
data/policy_packs.yaml isn't only deciding which packs — it's also carrying everything the page generator needs and the API doesn't have: URL slug, page title, H1, meta description, menu parent, weight, and the framework grouping that puts cis-aws / cis-azure / cis-google-cloud under one "CIS 8.1" section. Worth noting displayName is "" on both records Dan pasted, so we can't even derive titles from the listing.
So the file stays as the editorial layer and stops being the gate. source: pulumi now decides membership, checked both ways:
- a documented pack that isn't a product → fail (this is what shipped in
cd00d9a2) - a product nobody documented → fail (new in
e7794493)
The second one is the same bug this PR was opened for, just pointing the other way: we shipped ISO 27001 for three clouds and the docs said AWS. Under the old design a new product pack could ship and docs would stay silent indefinitely. Now the nightly fails until someone either adds an entry or records why not.
"Why not" is now a machine-readable undocumented: list rather than a comment, holding the two genuine omissions — aws-organizations-tag-policies (hand-written prose page, not a policy listing) and cis-kubernetes-google-cloud (the stale publish superseded by cis-kubernetes-gcp). If either drops out of that list the build fails, so they're enforced rather than remembered.
On your other message — yes, publisher: pulumi was the trap. It's "pulumi" for every pack in the org including the private ones, so it would have admitted exactly what we were trying to exclude. The check uses source only, with a comment saying don't switch it, since publisher is the more natural-looking field.
Last night's run failed exactly as expected: pack "hitrust-awsnative" is in data/policy_packs.yaml but is not published to the "pulumi" org (run). master still carries the entry you deleted from the org, so that repeats nightly at 07:30 UTC until this merges.
Generated by Claude Code
jkodroff
left a comment
jkodroff
left a comment
There was a problem hiding this comment.
Approving, since this fixes the most important mistake, but it's not a full solution.
|
@jkodroff I'm rolling in the allowlist idea |
Turns the rule in data/policy_packs.yaml from prose into a check. `/api/orgs/<org>/ policypacks`, which this script already reads, returns everything the org holds -- products and privately published packs alike, indistinguishably. That is how the private hitrust-awsnative pack got a public reference page (#21055). The registry listing carries the distinction in `source`: "pulumi" for a published product, "private" otherwise. Every allowlisted pack must now come back as a product or the job fails naming the offenders and their actual source. Verified against real records from @danbiwer, including the two failure modes behind this incident: a privately published pack, and a pack absent from the listing entirely. Deliberately not `publisher`: that is "pulumi" for everything in our org, so it would admit private packs too. `source` is set by an admin command rather than by whoever publishes, so it cannot be set by an engineer testing something -- which is what makes it safe to fail the build on. Fails closed. An unreadable or changed response stops the job rather than falling back to publishing unverified pages. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BvCEyksviZ5M9vNaZvAsdz
github-actions
Bot
added
review:outstanding-issues
The step I added ran before anything was installed, and this module required js-yaml at load, so the suite died on MODULE_NOT_FOUND before a single case ran. Installing first is the wrong fix: `make ensure` begins with `clean`, so anything installed ahead of `make ci_pull_request` is thrown away. Moving the tests after the build would report a failure only once the site had already deployed. js-yaml is used in exactly one place -- reading the allowlist in main() -- and none of the exported helpers need it, so requiring it there keeps the module importable with no node_modules. Verified by hiding node_modules/js-yaml and running the suite the way CI does: 16/16. main() still parses the allowlist. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BvCEyksviZ5M9vNaZvAsdz
github-actions
Bot
added
review:stale
|
@claude #update-review One commit since the last refresh, The step ran before anything was installed and this module required Installing first would have been wasted work ( Verified the way CI does it: hid No other findings affected; the worklist was clean before this push and the change touches only the require site and a workflow comment. Generated by Claude Code |
github-actions
Bot
added
review:in-progress
|
🤖 Review updated on @CamSoper's request. |
github-actions
Bot
added
review:no-blockers
…es-gcp Two comment-only fixes, both about the file staying honest about itself. Hoisting the integrity checks last round moved their code but carried the reverse-direction preamble with it, leaving a paragraph about `undocumentedProducts` sitting above `staleExemptions` -- which is the opposite check. Moved it back to the code it describes. Comment-code drift is the failure this whole change exists to prevent, so introducing some while fixing it is worth undoing. `cis-kubernetes-gcp` had nowhere to live after ea95624 stopped documenting it: not in `sections:`, not in `undocumented:`, and not in the do-not-document list whose stated job is stopping an audit from re-adding a name. It is the name most likely to be re-added by mistake, being the plausible-looking half of the pair and the exact mistake just fixed, and no check can catch it -- a private pack nothing references is invisible to both directions of the gate. Now recorded with the reason. The header claimed all listed packs come back source: "private", which glossed hitrust-awsnative: it is not private, it is gone from the org. Reworded to say which 13 are private and to name the exception. Verified against the live listing: 13 private, all 13 accounted for, none unlisted. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BvCEyksviZ5M9vNaZvAsdz
github-actions
Bot
added
review:stale
|
@claude #update-review Both fixed in
Re-ran the real fetch against the live API afterward: exit 0, and Worklist is clean. Merging once this comes back. Generated by Claude Code |
github-actions
Bot
added
review:in-progress
|
🤖 Review updated on @CamSoper's request. |
github-actions
Bot
added
review:no-blockers
…l-draft-context-uc1jpa
github-actions
Bot
added
review:stale
5 participants
The list that decides which pre-built policy packs get a documentation page,
data/policy_packs.yaml, is hand-maintained and was never checked against what Pulumi actually publishes. It had drifted in both directions: frameworks we ship for three clouds documented as AWS-only, and packs published privately to the org carrying public reference pages.This corrects the drift and adds a check so it can't recur silently.
What's here
/docs/reference/pre-built-policy-packs/have been correct all along; the hand-maintained summary pages understated them. A reader comparing the two would conclude we ship less than we do.hitrust-awsnativefrom the table, the AWS integrations page, and the allowlistPolicy Packs - Update Policiesjob currently fails onmaster— merging this clears it.cis-kubernetes-gcptocis-kubernetes-google-cloudpolicy-packs-internalpublishes undercis-kubernetes-gcp, but that name is private, so the documented page was built from a pack customers cannot enable. The public pack iscis-kubernetes-google-cloud. Confirmed with @danbiwer, who is renaming it upstream so releases land on the public name. Both carry the same 50 policies and the slug is unchanged, so no page content or URL moves.scripts/fetch-policy-packs.js/api/orgs/{org}/policypacks— which this script already reads — returns products and privately published packs indistinguishably. The registry listing distinguishes them withsource: "pulumi", so the build can now enforce what a comment used to ask for.scripts/run-unit-tests.shyarn --cwd components testagainst a directory that no longer exists — so it is repaired and wired intomake test.setting-up-for-success.mdadvisory/mandatory/remediate.Coverage corrections
How the check works
Every entry in
data/policy_packs.yamlmust come back from/api/registry/policypacks?orgLogin={org}withsource: "pulumi", and every pack that does must have either a page or a recorded exemption. Both directions fail the build.publisher. It is"pulumi"for every pack in the org, including the private ones, so filtering on it would admit exactly what this excludes. It is the field a reasonable person reaches for first, so there is a comment saying not to.sourceis set by an admin command, not by whoever publishes the pack, which is what makes it safe to fail a build on.undocumented:records products that deliberately get no generated page. Bothpack:andwhy:are required, so the list cannot be used to quietly silence the check.data/policy_packs.yamlremains the editorial layer — slugs, titles, meta descriptions, menu placement, framework grouping. The registry carries none of that (displayNameis empty on every record). What changed is that it is no longer the source of truth for which packs are documented.Verification
Run against the live listing for the
pulumiorg: 36 packs, 23 withsource: "pulumi". All 22 documented packs plus the one recorded exemption account for exactly those 23, with no products left undocumented.make lint,make test-unit(15/15), andprettier --checkpass.Relationship to #21120
#21120 adds an Edition column to the same table and per-section
feature:keys todata/policy_packs.yaml. That PR notes it left out the packs missing from the table pending confirmation they are GA — the registry query above is that confirmation, and those packs are what this PR adds. The two conflict onpre-built-packs.mdandpolicy-packs/_index.md; this branch rebases onto #21120 once it lands, and the resolved table is this PR's coverage with that PR's column.Follow-ups, not in this PR
content/docs/iac/comparisons/*.md, andcontent/what-is/what-is-policy-as-code.md(L199).