ci(deploy): build once on GitHub, secret-scan, push to GHCR; server pulls by SHA

.dockerignore

@@ -39,6 +39,23 @@ tests/
 # Logs
 *.log
 
-# Local env
-.env
-.env.local
+# Secrets, at any depth. The images are published to GHCR (images.yml), so a file
+# that reaches a build context can reach the internet. `.env` alone matched only
+# the root one: apps/admin/.env* went into the admin image (and `vite build` reads
+# .env.production), and `COPY backend/src/` + `dotnet publish` shipped any
+# untracked appsettings.<Env>.json into /app. A clean CI checkout has none of these;
+# a server or laptop checkout can. scripts/scan-image-secrets.sh checks the result.
+**/.env
+**/.env.*
+!.env.example
+**/appsettings.*.json
+**/*service-account*.json
+**/*.pem
+**/*.key
+**/*.p12
+**/*.pfx
+
+# Local .NET build output. COPY backend/src/ would otherwise carry a laptop's
+# bin/obj into the build stage, and all of it into the single-stage migrator.
+**/bin/
+**/obj/

.github/docker-compose.ci.yml

@@ -1,8 +1,10 @@
 # CI-only overlay: Docker layer cache in the GitHub Actions cache. Used by the
 # `docker` and `e2e` jobs in ci.yml as
 #   docker compose -f docker-compose.yml -f .github/docker-compose.ci.yml build
-# and by nothing else. The images keep compose's own names, so a plain
-# `docker compose up -d` afterwards finds them and does not rebuild.
+# and by images.yml, which publishes the same images to GHCR (and so keeps this
+# cache warm for pull requests). The images keep the names docker-compose.yml
+# gives them, so a plain `docker compose up -d` afterwards finds them and does
+# not rebuild.
 #
 # One scope per image: a shared scope would let each image's cache manifest
 # overwrite the last. Both jobs build the same images from the same args, so they
@@ -30,3 +32,7 @@ services:
     build:
       cache_from: ["type=gha,scope=compose-ssg-worker"]
       cache_to: ["type=gha,scope=compose-ssg-worker,mode=max,ignore-error=true"]
+  mcp-server:
+    build:
+      cache_from: ["type=gha,scope=compose-mcp-server"]
+      cache_to: ["type=gha,scope=compose-mcp-server,mode=max,ignore-error=true"]

.github/workflows/deploy.yml

@@ -43,9 +43,27 @@ jobs:
     with:
       scope: merged-tree
 
+  # Builds and pushes the images for this commit (or the rollback commit) on a
+  # GitHub-hosted runner, in parallel with `ci`. A job in the same run rather than
+  # a separate workflow the server polls for: `needs` gives the exact SHA with no
+  # timeout to guess and no race, and the rollback input reaches it directly.
+  images:
+    uses: ./.github/workflows/images.yml
+    permissions:
+      contents: read
+      packages: write
+    with:
+      ref: ${{ github.event.inputs.rollback_commit }}
+
   deploy:
-    needs: ci
+    needs: [ci, images]
+    # A failed `images` must not block a deploy: the server builds instead, as it
+    # always did. A failed `ci` still does.
+    if: ${{ !cancelled() && needs.ci.result == 'success' }}
     runs-on: self-hosted
+    permissions:
+      contents: read
+      packages: read
     # 2m typical, 30m at worst.
     timeout-minutes: 60
 
@@ -71,8 +89,11 @@ jobs:
           if [ -n "${{ github.event.inputs.rollback_commit }}" ]; then
             git checkout ${{ github.event.inputs.rollback_commit }}
           else
+            # This run's own commit, not main's tip: the images job built exactly
+            # github.sha, and a newer merge queued behind us gets its own run.
+            # ff-only: re-running an older deploy fails loudly — use rollback_commit.
             git checkout main
-            git pull origin main
+            git merge --ff-only ${{ github.sha }}
           fi
 
       # The frontend that reaches production is built by this job, and until now
@@ -304,18 +325,61 @@ jobs:
         # `runtime: nvidia` + GPU device reservation to the ollama service.
         # CI runners don't have nvidia-container-toolkit, so it's NOT in
         # the default compose file (that would break the ci/docker job).
+        #
+        # Images come from GHCR, built once by the `images` job for this exact
+        # commit. If any pull fails (job failed, registry down, a commit from before
+        # images were published) the server builds them itself, as it used to.
+        env:
+          GHCR_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          GHCR_USER: ${{ github.actor }}
         run: |
           cd $PROJECT_DIR
           # Stamp the deployed commit into the images as SENTRY_RELEASE (build arg →
           # ENV). Without a release Sentry can't do regression detection or tell you
           # which deploy an issue started with. .dockerignore excludes .git, so the
-          # build arg is the only way in.
-          export GIT_SHA=$(git rev-parse --short HEAD)
+          # build arg is the only way in. --short=12, the same as images.yml, so a
+          # pulled and a fallback-built image report the same release.
+          GIT_SHA=$(git rev-parse --short=12 HEAD)
+          IMAGE_TAG=$(git rev-parse HEAD)
+          IMAGE_REGISTRY=ghcr.io/${{ github.repository_owner }}/textstack
+          export GIT_SHA IMAGE_TAG IMAGE_REGISTRY
           echo "Deploying release $GIT_SHA"
           # --profile mcp brings up the remote MCP server (mcp-server) in prod.
           # It's profile-gated so CI's bare `compose up` (docker/e2e jobs) never
           # builds/runs it; prod opts in here. nginx /mcp is synced below.
-          docker compose --profile mcp -f docker-compose.yml -f docker-compose.gpu.yml up -d --build
+          COMPOSE="docker compose --profile mcp -f docker-compose.yml -f docker-compose.gpu.yml"
+
+          PULLED=false
+          if grep -q 'IMAGE_TAG' docker-compose.yml; then
+            # The token goes into a throwaway config dir for these pulls only, never
+            # into ~/.docker on the server. It expires with this job anyway. Once the
+            # packages are public the login is unnecessary but harmless.
+            DOCKER_CONFIG=$(mktemp -d) && export DOCKER_CONFIG
+            if echo "$GHCR_TOKEN" | docker login ghcr.io -u "$GHCR_USER" --password-stdin; then
+              PULLED=true
+              for s in migrator api worker admin ssg-worker mcp-server; do
+                timeout 900 docker pull "$IMAGE_REGISTRY-$s:$IMAGE_TAG" || { PULLED=false; break; }
+              done
+            fi
+            rm -rf "$DOCKER_CONFIG"; unset DOCKER_CONFIG
+          fi
+
+          if [ "$PULLED" = true ]; then
+            echo "Deploying pulled images $IMAGE_REGISTRY-*:$IMAGE_TAG"
+            $COMPOSE up -d --no-build
+          else
+            echo "::warning::Images for $IMAGE_TAG not pulled — building on the server"
+            $COMPOSE up -d --build
+          fi
+
+          # A hand-run `docker compose up -d <svc>` on the server (e.g. after editing .env)
+          # has no IMAGE_TAG and resolves `textstack-<svc>:latest`. Point that at what is
+          # running now, or it would silently start an older image.
+          if grep -q 'IMAGE_TAG' docker-compose.yml; then
+            for s in migrator api worker admin ssg-worker mcp-server; do
+              docker tag "$IMAGE_REGISTRY-$s:$IMAGE_TAG" "textstack-$s:latest"
+            done
+          fi
 
       # Keep the local model current. A no-op when the manifest hasn't changed (seconds); the model
       # was otherwise pulled once by hand and never again. Non-fatal: a failed pull keeps the model
@@ -530,4 +594,12 @@ jobs:
           systemctl --user restart quality-poller && echo "Quality poller restarted" || echo "Quality poller restart FAILED (run: make quality-poll-setup if not installed)"
 
       - name: Cleanup old images
-        run: docker image prune -f
+        run: |
+          # Every deploy pulls a new SHA tag, and a tagged image is never "dangling", so
+          # prune alone would keep them all. Drop every published tag but this deploy's.
+          # In-use images refuse removal, which is the point.
+          KEEP=$(git -C "$PROJECT_DIR" rev-parse HEAD)
+          docker image ls --format '{{.Repository}}:{{.Tag}}' \
+            | grep '^ghcr\.io/${{ github.repository_owner }}/textstack-' | grep -v ":$KEEP\$" \
+            | xargs -r docker rmi || true
+          docker image prune -f

.github/workflows/images.yml

@@ -0,0 +1,122 @@
+name: Images
+
+# Build once, deploy many. Builds the compose images on a GitHub-hosted runner,
+# scans them for secrets, and pushes them to GHCR as
+#   ghcr.io/<owner>/textstack-<service>:<full commit sha>
+# deploy.yml calls this next to its CI job, and the server pulls those exact tags
+# instead of building (it still builds itself if the pull fails).
+#
+# It replaced ci-cache-warm.yml, which built the same images on main only to warm
+# the layer cache and threw them away. Same overlay and cache scopes, so pull
+# requests still start warm.
+#
+# The packages are meant to be PUBLIC (free). That is only safe if nothing secret
+# is baked in, so the scan below gates the push: every .env value is a canary, and
+# a canary, a private key or a token pattern anywhere in an image fails the job
+# before anything leaves the runner.
+on:
+  workflow_call:
+    inputs:
+      ref:
+        description: Commit to build. Empty = the commit that triggered the run.
+        type: string
+        default: ''
+  workflow_dispatch:
+    inputs:
+      ref:
+        description: Commit to build (empty = the selected branch head)
+        required: false
+
+env:
+  FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: 'true'
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+    # ~12m cold, a few minutes warm, plus the push and scan.
+    timeout-minutes: 40
+    permissions:
+      contents: read
+      packages: write
+    env:
+      IMAGE_REGISTRY: ghcr.io/${{ github.repository_owner }}/textstack
+      # Every service with a `build:` in docker-compose.yml. deploy.yml pulls the same list.
+      SERVICES: migrator api worker admin ssg-worker mcp-server
+    steps:
+      - uses: actions/checkout@v7
+        with:
+          ref: ${{ inputs.ref || github.sha }}
+
+      - name: Resolve tag
+        id: tag
+        run: |
+          echo "IMAGE_TAG=$(git rev-parse HEAD)" >> "$GITHUB_ENV"
+          # Same value the server stamps on a fallback build — one Sentry release per commit.
+          echo "GIT_SHA=$(git rev-parse --short=12 HEAD)" >> "$GITHUB_ENV"
+          # A rollback to a commit from before this workflow existed: its compose file has
+          # no `image:` entries, so the server builds it, as it always did.
+          if grep -q 'IMAGE_TAG' docker-compose.yml; then
+            echo "publish=true" >> "$GITHUB_OUTPUT"
+          else
+            echo "::notice::$(git rev-parse --short HEAD) predates published images — the server will build it"
+            echo "publish=false" >> "$GITHUB_OUTPUT"
+          fi
+
+      - uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
+        if: steps.tag.outputs.publish == 'true'
+
+      - name: Log in to GHCR
+        if: steps.tag.outputs.publish == 'true'
+        run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u "${{ github.actor }}" --password-stdin
+
+      # A rollback to a commit that was deployed before is already in the registry.
+      - name: Already published?
+        id: exists
+        if: steps.tag.outputs.publish == 'true'
+        run: |
+          for s in $SERVICES; do
+            if ! docker buildx imagetools inspect "$IMAGE_REGISTRY-$s:$IMAGE_TAG" >/dev/null 2>&1; then
+              echo "all=false" >> "$GITHUB_OUTPUT"; exit 0
+            fi
+          done
+          echo "All images for $IMAGE_TAG already published"
+          echo "all=true" >> "$GITHUB_OUTPUT"
+
+      - name: Free disk space
+        if: steps.exists.outputs.all == 'false'
+        run: |
+          sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /opt/hostedtoolcache
+          docker system prune -af
+
+      - uses: crazy-max/ghaction-github-runtime@04d248b84655b509d8c44dc1d6f990c879747487 # v4.0.0
+        if: steps.exists.outputs.all == 'false'
+
+      # Every value a canary. Compose interpolates .env into build args, so if anyone
+      # ever forwards a real variable into a build (`args: KEY: ${OPENAI_API_KEY}`),
+      # the canary lands in the image and the scan fails.
+      - name: Create canary .env
+        if: steps.exists.outputs.all == 'false'
+        run: |
+          sed -n 's/^\([A-Z_][A-Z0-9_]*\)=.*/\1=CANARY_SECRET_\1/p' .env.example > .env
+          for v in SENTRY_DSN GOOGLE_LEGACY_CLIENT_IDS GUEST_SESSION_PERMIT_LIMIT USER_LOGIN_PERMIT_LIMIT \
+                   CLIP_PERMIT_LIMIT ACCOUNT_DELETE_PERMIT_LIMIT GUEST_DAILY_ENRICHMENT_CAP ENABLE_TEST_AUTH; do
+            echo "$v=CANARY_SECRET_$v" >> .env
+          done
+          cat .env
+
+      - name: Build
+        if: steps.exists.outputs.all == 'false'
+        run: |
+          # shellcheck disable=SC2086
+          docker compose --profile mcp -f docker-compose.yml -f .github/docker-compose.ci.yml build $SERVICES
+
+      - name: Secret scan (gates the push)
+        if: steps.exists.outputs.all == 'false'
+        run: |
+          # shellcheck disable=SC2046
+          scripts/scan-image-secrets.sh $(for s in $SERVICES; do echo "$IMAGE_REGISTRY-$s:$IMAGE_TAG"; done)
+
+      - name: Push
+        if: steps.exists.outputs.all == 'false'
+        run: |
+          for s in $SERVICES; do docker push "$IMAGE_REGISTRY-$s:$IMAGE_TAG"; done

CHANGELOG.md

@@ -28,6 +28,7 @@ the archive; if it broke production, it belongs in `docs/incidents/`. See
 - **Reader** — highlights on the phone now paint on their words anywhere in a chapter; since April they were drawn `scrollY` px above (off screen) below the first screen, because the overlay svg moved with the page and was also shifted by the scroll — mobile
 - **Privacy** — policy rewritten to match the code: uploads send title, author and opening text to OpenAI and title/author to Open Library, mobile creates a guest account on open, models, unlinked AI records, mobile Sentry, Expo updates, Edge TTS, MCP connectors, 30-day caches, 14-day logs, ~90-day encrypted backups; delete page states what remains; nginx access logs drop query strings (`/api/tts?text=`, search); Data Safety runbook re-derived — web, mobile, infra, docs · [details](docs/changelog-archive/2026-H2.md#2026-10-06-privacy-policy-matches-code)
 - **Privacy** — policy rewritten to match the code: uploads send title, author and opening text to OpenAI and title/author to Open Library, mobile creates a guest account on open, models, unlinked AI records, mobile Sentry, Expo updates, Edge TTS, MCP connectors, 30-day caches, 14-day logs, ~90-day encrypted backups; delete page states what remains; nginx access logs drop query strings (`/api/tts?text=`, search); container logs rotated (json-file 10m×5); Data Safety runbook re-derived — web, mobile, infra, docs · [details](docs/changelog-archive/2026-H2.md#2026-10-06-privacy-policy-matches-code)
+- **Deploy** — build once, deploy many: `images.yml` (replaces `ci-cache-warm.yml`) builds the compose images on GitHub, secret-scans them (canary `.env`, keys, token patterns; scan gates the push) and pushes `ghcr.io/mrviduus/textstack-<service>:<sha>`; the server pulls that SHA and builds itself only if the pull fails; rollback pulls (or publishes) the rollback SHA; `.dockerignore` now drops nested `.env*`, keys, service accounts, `appsettings.*.json`, `bin/`/`obj/` — infra
 - **Reader** — a font change at open, during the restore, no longer leaves a window where back or backgrounding saves the chapter top: once a stale restore ack arrives, position reports stop opening the write gate, and only the re-ask's ack or the 4 s deadline opens it; a truly lost ack still opens by position — mobile
 - **Reader (R4)** — a rebuild or font change during the open restore keeps the saved place (non-default typography no longer reopens at the top and saves 0); typography changed before the page loads is applied, also after "Read as text"; a PDF reopens on the device page inside the chapter, not the chapter start; the foreground newer-position check runs after a background renderer kill; restore acks carry the landing position, so a tiny restore is not booked as reading; reflow/rebuild acks never reset the "has the reader moved" baseline — mobile · [ADR-019](docs/01-architecture/adr/ADR-019-reader-position-rules.md) · [details](docs/changelog-archive/2026-H2.md#2026-10-06-reader-r4-and-adr-019)
 - **Refactor** — web `ReaderPage.tsx` (898 → 429) and `ReaderHighlights.tsx` (661 → 415) split by job into 8 hooks and 6 components, no behaviour change; source guards read the page plus every split-out file — web

CLAUDE.md

@@ -490,7 +490,7 @@ That single command builds the AAB and pushes it to Internal Testing. Service ac
 
 **GitHub Actions workflows** (`.github/workflows/`):
 - **ci.yml** — runs on PR + push to main. Jobs: backend (build, lint, migrations, search tests), frontend (web + admin build), docker (integration tests), e2e (Playwright)
-- **deploy.yml** — self-hosted runner on server. Pre-deploy backup → git pull → frontend build → docker compose up → health checks → SSG content check → image cleanup. Full SSG rebuild only with the `rebuild_ssg` input; otherwise nightly in backup.yml
+- **deploy.yml** — `images` job (GitHub-hosted, `images.yml`) builds every image once, gates it on `scripts/scan-image-secrets.sh`, pushes `ghcr.io/mrviduus/textstack-<svc>:<sha>`; then the self-hosted runner: pre-deploy backup → checkout this run's SHA → frontend build → pull images by SHA (`up -d --no-build`; a failed pull falls back to building on the server) → health checks → SSG content check → image cleanup. Secrets live only in the server's `.env` at runtime, never in an image. Full SSG rebuild only with the `rebuild_ssg` input; otherwise nightly in backup.yml
 - **backup.yml** — daily at 3 AM UTC. DB dump + storage tar.gz, keeps 5 newest of each
 - **health-check.yml** — every 5 min. Checks API + both frontends