Skip to content

secrets.md

Latest commit

 

History

History
145 lines (94 loc) · 11.1 KB

secrets.md

File metadata and controls

145 lines (94 loc) · 11.1 KB
Raw

Shared credentials: super.env in Cloudflare Secrets Store

~/super.env on the primary VPS holds every shared fleet credential. That VPS stays the one place the file is edited. Cloudflare Secrets Store keeps a copy, and a new agent host fetches it through a private Worker. The fetched file carries its own usage notes as comments.

Never paste a value from this file into a repository, issue, PR, log, or chat. Compare copies by sha256sum only.

What lives where

Piece Where Notes
Source of truth ~/super.env on the primary VPS, mode 600 Edit here, then push.
The whole file Account Secrets Store (default_secrets_store), secrets super_env_<hash>_0, super_env_<hash>_1, …, comment super.env, scope workers The file's base64, cut into chunks of at most 65,536 characters. <hash> is the first 12 hex digits of the file's sha256. The comments hold retired credentials, so they stay a secret too.
Client id for the Worker's own check Secret FLEET_SECRETS_ACCESS_CLIENT_ID, comment super.env, scope workers A copy of that variable's last assignment, without one surrounding pair of matching quotes.
Single secrets Secrets in default_secrets_store that scripts/put-secret.sh stored, comment put-secret.sh, scope workers Not part of super.env. The Worker serves one only when FLEET_SECRETS_READABLE is its name.
Fetch endpoint Worker fleet-secrets on the custom domain host names in scripts/stow-secrets.sh No workers.dev or preview URL. Answers with Cache-Control: no-store.
Gate Cloudflare Access app fleet-secrets, one policy: Service Auth for service token fleet-secrets-fetch The Worker also verifies the Access JWT: issuer, audience, expiry, signature, and the token's client id. Anything else gets 403.
Fetch credential FLEET_SECRETS_ACCESS_CLIENT_ID, FLEET_SECRETS_ACCESS_CLIENT_SECRET at the end of ~/super.env Expires one year after creation.

apps/fleet-secrets/split.jq defines the split. The Worker joins its bindings super_env_0, super_env_1, … in order and decodes them, which gives the file byte for byte.

Limits: a secret holds at most 65,536 bytes, and an account holds at most 100 secrets during the Secrets Store beta. One chunk holds 49,152 bytes of the file, so the secret count grows by one per 48 KiB of file, not per variable. A push needs room for the old and the new chunks together, and makes that room itself when the store is full (see below).

Push after editing super.env

On the VPS, from this repository:

scripts/stow-secrets.sh

It reads CLOUDFLARE_ACCOUNT_ID and the account token CF_API_TOKEN_GLOBAL from the file itself. It creates the chunks for the current file and updates FLEET_SECRETS_ACCESS_CLIENT_ID, then redeploys the Worker bound to them, then turns workers.dev and preview URLs off. It refuses to overwrite a same-named secret whose comment is not super.env. Running it twice leaves the same state.

A changed file gets new chunk names, so a push never changes a chunk that a running Worker reads. After the redeploy, the script waits 30 seconds for every edge to run the new Worker, then deletes every secret with comment super.env that the new Worker does not bind: older chunks and the secrets of the earlier one-secret-per-variable layout. It never deletes a secret with another comment.

If the store has no room for the new chunks (secret count plus chunks to create above 100), the script deletes just enough of those replaced super.env secrets before it creates the chunks, then continues as above. The running Worker then loses some bindings until the redeploy finishes, so fetch-secrets.sh can fail for a few seconds. This happens whenever the store lacks room, normally only once, on the move from the one-secret-per-variable layout. The fetch works again after the redeploy.

Put one secret

On the VPS, from this repository, pipe the value on stdin:

scripts/put-secret.sh EXAMPLE_ADMIN_PASSWORD <value-file

It creates or updates exactly the secret named EXAMPLE_ADMIN_PASSWORD in the store named default_secrets_store. A new secret gets comment put-secret.sh and scope workers; an update changes only the value. It reads the value only from stdin and drops one trailing newline. It refuses a terminal on stdin, an empty value, a second argument, and a name with characters other than letters, digits and _. It never prints the value. It uses CLOUDFLARE_ACCOUNT_ID and CF_API_TOKEN_GLOBAL from ~/super.env, as stow-secrets.sh does.

It changes no other secret, deletes nothing, and does not deploy the Worker. It refuses a secret with comment super.env, because stow-secrets.sh owns those. stow-secrets.sh never deletes or overwrites a secret with another comment, so a later push keeps it.

Let hosts read it

The Worker serves a single secret at GET /NAME only when it binds that secret as read_NAME. It answers 403 without a valid Access JWT, and 404 for every name it does not bind this way, which includes the chunks and FLEET_SECRETS_ACCESS_CLIENT_ID.

  1. Set FLEET_SECRETS_READABLE to the name in ~/super.env on the VPS. It takes one name only, not a list:

    FLEET_SECRETS_READABLE=EXAMPLE_ADMIN_PASSWORD

  2. Deploy the binding, as a separate step:

    scripts/stow-secrets.sh

    This is the only step that changes what the live Worker can read. It stops before any change when the value is not one name, or the name is missing from the store or has comment super.env. No GitHub workflow deploys the fleet-secrets Worker: harbor.yml deploys only the crewship site Worker.

  3. On a host with the fetch credential (see below), print the value:

    scripts/fetch-secrets.sh EXAMPLE_ADMIN_PASSWORD

    It writes only to stdout and leaves ~/super.env unchanged.

To stop serving the secret, remove FLEET_SECRETS_READABLE and run scripts/stow-secrets.sh again.

Fetch on a new host

  1. On the VPS, copy the two lines FLEET_SECRETS_ACCESS_CLIENT_ID=… and FLEET_SECRETS_ACCESS_CLIENT_SECRET=… from ~/super.env. Copy them by hand, over SSH or a password manager; never through chat, a ticket, or a repository.

  2. On the new host, write them to ~/.config/fleet-secrets.env and restrict it:

    mkdir -p ~/.config && chmod 700 ~/.config
install -m 600 /dev/null ~/.config/fleet-secrets.env
"${EDITOR:-nano}" ~/.config/fleet-secrets.env

  3. From a checkout of this repository, run:

    scripts/fetch-secrets.sh

    It writes ~/super.env at mode 600 through a temp file and an atomic rename, then prints its sha256. It must equal sha256sum ~/super.env on the VPS.

The script reads the two values only from ~/.config/fleet-secrets.env. It refuses that file unless it is mode 600, and it never takes the values as arguments or environment variables. On any HTTP status other than 200, the existing ~/super.env stays unchanged. FLEET_SECRETS_URL overrides the URL.

A fetched copy is read-only in practice: edits made on it are overwritten by the next fetch. Make edits on the VPS and push.

Revoke if a host is lost

  1. Cloudflare dashboard → Zero Trust → Access → Service credentials → Service Tokens → fleet-secrets-fetch → Delete. Fetching stops immediately for every host.
  2. Create a new service token, and replace the token_id in the fleet-secrets app's only policy.
  3. Replace the two FLEET_SECRETS_ACCESS_* lines in ~/super.env on the VPS with the new token's client id and secret, then run scripts/stow-secrets.sh. The Worker only accepts the client id stored in FLEET_SECRETS_ACCESS_CLIENT_ID.
  4. Copy the new lines to the remaining hosts.

Revoking the fetch token does not revoke what the lost host already fetched. It holds every credential in the file, so treat each one as exposed, and tell the owner.

Renew the token the same way before it expires; the Access dashboard shows the expiry date.

GitHub App

The fleet's read-only GitHub calls use GitHub App installation tokens, so they draw on app rate limits and leave the user token's 5,000 requests an hour for what an app cannot do.

Each repository the owner owns has its own GitHub App, named i098-<repo> and installed on that repository only, so each repository has its own installation rate limit. ~/super.env holds GITHUB_APP_<REPO>_ID, GITHUB_APP_<REPO>_INSTALLATION_ID and GITHUB_APP_<REPO>_KEY_PATH for each, with <REPO> in upper case and every other character as _. The key path names a mode 600 file in ~/.config/crewship/gh-apps/, outside super.env; copy it to another host by hand, as for the fetch credential above. A repository without these three settings uses the user login as before.

gh-read-token [REPO] prints that repository's token; only i098/… repositories have apps, and any other owner gets the user token at once with no app lookup. It signs a short-lived JWT with openssl, asks GitHub for an installation token, and caches it in ~/.cache/crewship/gh-read-token/i098-<repo>.json (mode 600) until 5 minutes before it expires. gh-read-token run gh api … picks the repository from the repos/owner/name in the command, or from the current directory's origin remote, runs the read with GH_TOKEN set to that token, and runs it again as the user when the repository has no app or the app token fails. A 304 answer to a conditional request counts as success and is not replayed. The command runs with GH_READ_TOKEN_ACTIVE=1, so the gh guard wrapper does not start the helper a second time.

Nothing needs a manual renewal. A GitHub App private key has no expiry date (the key page of each app shows none). The installation token lasts one hour, and the helper mints a new one from the key whenever fewer than 5 minutes remain. Revoke a key only if it leaks.

Each app has read access to contents, pull requests, checks, issues, actions, statuses, metadata and administration, and write access to contents and pull requests. These callers read through it:

  • the Cloudflare runner dispatch (workflow runs and jobs) and the CI pool runner watch (runner list);
  • the Herdr sidebar's PR lookup;
  • every gh api repos/… read that goes through the gh guard.

These calls stay on the user login, because an app cannot make them or has no permission for them: runner registration and removal (generate-jitconfig, DELETE …/actions/runners/…), user notifications (fm-gh-replies), user-owned Projects (github-board.py), gh api user, gh-axi and GraphQL calls, and issue writes. Repositories of an organization need the organization owner to install an app.

One-time setup, for rebuilding

Create these once per Cloudflare account, and again only if they are lost:

  1. Access app fleet-secrets: self-hosted, on the Worker's custom domain, session 15 minutes, hidden from the App Launcher, no identity providers. Its only policy: decision Service Auth, include only service token fleet-secrets-fetch.
  2. Run scripts/stow-secrets.sh. It reads the team domain and the app's Application Audience tag from the Access API on every push and binds them to the Worker.
  3. Add the Worker custom domain only after the Access app exists.