feat(onchain-analytics): prepare safe additive schema commissioning

projects/onchain-analytics/README.md

@@ -130,7 +130,7 @@ The pipeline and dbt are independent — the pipeline writes raw tables, dbt rea
 | [`gd_dbt/`](gd_dbt/) | dbt project — all warehouse models, tests, docs, macros |
 | [`pipeline-v5/`](pipeline-v5/) | The ingestion pipeline (TypeScript). The only one. Runbook in its own README |
 | [`warehouse/L1/`](warehouse/L1/) | Raw table DDL (pipeline-written tables, dbt *sources*) |
-| [`scripts/`](scripts/) | L1 bootstrap script (`deploy-warehouse.ps1`) |
+| [`scripts/`](scripts/) | Explicitly allowlisted L1 migration helper and sandbox validator |
 | [`contracts/`](contracts/) | ABI files, deployment block numbers, contract reference |
 | [`docs/`](docs/) | System documentation, data model, operations guide, governance |
 
@@ -142,7 +142,8 @@ The pipeline and dbt are independent — the pipeline writes raw tables, dbt rea
 
 - Node.js LTS (v20+)
 - Google Cloud SDK (`gcloud`, `bq`)
-- `gcloud auth application-default login` with BigQuery Job User + Data Editor on `gooddollar`
+- `gcloud auth application-default login` for metadata reads and labelled sandbox validation
+- Production L1 DDL and ingestion use separately approved impersonated identities; see [`03_OPERATIONS.md`](docs/03_OPERATIONS.md)
 - Python 3.9+ with dbt-bigquery (`pip install dbt-bigquery`)
 
 ### Run the warehouse
@@ -164,12 +165,16 @@ npx tsx src/index.ts daily    # Ingest from chain into the BigQuery raw tables
 npx tsx src/index.ts verify   # Reconcile what was ingested against the contracts
 ```
 
-### Bootstrap L1 raw tables (first time only)
+### Inspect the prepared L1 migration (plan-only)
 
 ```powershell
-.\scripts\deploy-warehouse.ps1
+.\scripts\deploy-warehouse.ps1 -Migration 09_CreateRawLogs_v1.sql
 ```
 
+This only prints the selected target. Validate migrations in a labelled sandbox first. Production
+execution requires separate approval, an explicit allowlisted migration, and service-account
+impersonation; see [`03_OPERATIONS.md`](docs/03_OPERATIONS.md).
+
 ---
 
 ## Documentation

projects/onchain-analytics/docs/03_OPERATIONS.md

@@ -97,16 +97,48 @@ nonzero.
 
 Two layers, two tools.
 
-### L1 raw tables — one-time bootstrap (PowerShell)
+### L1 raw tables -- allowlisted additive migrations
 
-The raw event tables (`BlockchainEvents.*`) are what the pipeline streams into. They are dbt
-*sources* (pipeline-written, dbt-read), not dbt models, so their DDL still lives in `warehouse/L1/`.
-Create them once:
+Raw tables are pipeline-written dbt sources, not dbt models. `scripts/deploy-warehouse.ps1` accepts
+one named migration from a fixed allowlist; it never scans `warehouse/L1/`. Its default mode only
+prints the target and migration name:
 
+```powershell
+.\scripts\deploy-warehouse.ps1 -Migration 09_CreateRawLogs_v1.sql
 ```
-.\scripts\deploy-warehouse.ps1        # creates the L1 raw tables
+
+Before any production change, validate the same migration files against a fresh labelled sandbox.
+From `pipeline-v5/`:
+
+```powershell
+node --import tsx ..\scripts\ops\validate-l0-migrations.mjs ..\..\_scratch\unit-07a-commissioning\sandbox-validation.json
 ```
 
+This sandbox check exercises the old `PipelineRuns` and `OracleReconciliation` shapes, repeats the
+migrations, checks their statement types and byte caps, verifies historical fixture rows remain,
+and proves the sandbox is absent after cleanup. It does not write production tables or ingest chain
+data.
+
+Production DDL is a separate operation and is not authorized by running the validator or plan mode.
+Only after separate approval of the exact migration and access list may the named administrator run
+one migration at a time:
+
+```powershell
+.\scripts\deploy-warehouse.ps1 -Migration 09_CreateRawLogs_v1.sql -Execute -AllowProduction `
+	-ImpersonateServiceAccount schema-commissioner@gooddollar.iam.gserviceaccount.com
+```
+
+The service-account address above is illustrative; replace it only with the approved identity. The
+helper refuses production execution without an explicit account. It sets
+`CLOUDSDK_AUTH_IMPERSONATE_SERVICE_ACCOUNT` only in the query child process's environment;
+persistent gcloud configuration and the caller's environment are never modified. Separate
+deployments therefore cannot overwrite each other's selected identity. The
+helper also requires a typed confirmation for `gooddollar.BlockchainEvents` and applies a 10 GiB
+per-job bytes cap. Stop if a live object differs from the measured schema baseline, an object that
+should be absent already exists, a migration returns `SCRIPT`, a legacy row count changes, or an
+effective permission is broader than the approved list. Never run `04_L0Contract_v3.sql`,
+`06_L0Contract_v4.sql`, or `07_RetireV3EventTables.sql` through this path.
+
 ### Staging, Semantic, Marts — dbt
 
 Everything above raw is managed by dbt. There are no numbered files to run by hand — dbt resolves
@@ -154,8 +186,8 @@ model/column docs with `dbt docs serve` (opens <http://localhost:8080>).
 |---|---|---|
 | `ENVIO_API_TOKEN is missing` | `pipeline-v5/.env` not created or empty | `cp pipeline-v5/.env.example pipeline-v5/.env` and fill in the token |
 | `Could not authenticate to Google` | gcloud ADC expired | `gcloud auth application-default login` again |
-| `Table not found: gooddollar.BlockchainEvents.…` | L1 DDL not run yet | Apply `warehouse/L1/04_L0Contract_v3.sql` |
-| `SCHEMA_MISMATCH: <table> has no column(s) …` | The pipeline writes a column the live table lacks | Apply the L0 contract. The pipeline refuses to write rather than corrupting a MERGE |
+| `Table not found: gooddollar.BlockchainEvents.…` | The prepared L1 schema migration has not been commissioned | Stop and check the approved migration list; do not run a historical contract file |
+| `SCHEMA_MISMATCH: <table> has no column(s) …` | The live schema differs from the runtime contract | Stop ingestion. Re-measure the schema and approve a new additive migration; do not recreate the table |
 | Run exits 1 with skipped chunks | HyperSync rate limiting or a timeout | Read `IngestionCoverage` for the exact ranges, then `backfill --from --to` over them |
 | `UNCONFIRMED EMPTY RANGE` | A range came back empty and no independent endpoint could confirm it | Not an error to clear by retrying. The watermark deliberately did not advance. Re-run when the endpoints recover |
 | `REORG SUSPECTED` | An existing key now sits under a different block hash | Delete and re-ingest that block range |

projects/onchain-analytics/pipeline-v5/README.md

@@ -86,10 +86,12 @@ cp .env.example .env          # then fill in ENVIO_API_TOKEN
 gcloud auth application-default login
 ```
 
-The BigQuery tables are created by the DDL in `../warehouse/L1/`, not by the pipeline. Apply
-`06_L0Contract_v4.sql` before ingesting anything. The pipeline refuses to write a column the
-live table does not have, and checks at startup that the bookkeeping tables carry the columns it
-is about to write, rather than failing part way into a backfill.
+The BigQuery tables are created by allowlisted migrations in `../warehouse/L1/`, not by the
+pipeline. The multi-statement `06_L0Contract_v4.sql` is a reference, not a deployment command.
+Before production commissioning, run the labelled-sandbox validator described in
+`../docs/03_OPERATIONS.md`. The pipeline refuses to write a column the live table does not have,
+and checks at startup that bookkeeping tables carry the columns it is about to write, rather than
+failing part way into a backfill.
 
 ### Two datasets, and why staging is not one of them
 

projects/onchain-analytics/scripts/README.md

@@ -5,14 +5,30 @@ One PowerShell helper remains. The warehouse (Semantic + Marts) is managed by **
 ## Prerequisites
 
 - Google Cloud SDK installed: <https://cloud.google.com/sdk/docs/install>
-- `gcloud auth application-default login` already run
-- Authenticated user has BQ Data Editor + Job User on the `gooddollar` project
+- `gcloud auth application-default login` already run for metadata reads and sandbox validation
+- Production schema changes require a separately authorized administrator; do not grant the ordinary
+	analytics identity raw-dataset write permissions
 
 ## What's here
 
 | Script | Purpose | When to run |
 |---|---|---|
-| [`deploy-warehouse.ps1`](deploy-warehouse.ps1) | Creates the **L1 raw tables** (`BlockchainEvents.*`) from the DDL in [`warehouse/L1/`](../warehouse/L1/). These are dbt *sources* (pipeline-written, dbt-read), not dbt models, so their bootstrap DDL still lives here. | Once after a clone, or if a raw table schema changes. |
+| [`deploy-warehouse.ps1`](deploy-warehouse.ps1) | Applies one named migration from a fixed allowlist. Default is plan-only; production execution requires `-Execute`, `-AllowProduction`, explicit service-account impersonation, and a typed confirmation. | Only after the exact migration and production access are separately approved. |
+
+The L1 SQL folder is not an execution queue. `04_L0Contract_v3.sql`, `06_L0Contract_v4.sql`,
+`07_RetireV3EventTables.sql`, and any unlisted file are refused. Run the labelled-sandbox migration
+validator from `pipeline-v5/` with `node --import tsx ..\scripts\ops\validate-l0-migrations.mjs ..\..\_scratch\unit-07a-commissioning\sandbox-validation.json`.
+
+Local regression checks from the project root, with no BigQuery jobs or credential acquisition:
+
+```powershell
+.\scripts\tests\deploy-warehouse.Tests.ps1
+node --test scripts/tests/validate-l0-migrations.test.mjs
+```
+
+The deployment checks verify overlapping child processes retain separate identities without
+changing persistent gcloud settings. The parser checks distinguish required and nullable columns
+without importing or executing the live sandbox validator.
 
 ## Everything else is dbt
 

projects/onchain-analytics/scripts/deploy-warehouse.ps1

@@ -1,34 +1,100 @@
 # deploy-warehouse.ps1
-# Creates the L1 raw event tables (BlockchainEvents.*) from the DDL in warehouse/L1/.
-# These are the tables pipeline-v5 writes into and dbt reads as sources; they are NOT managed
-# by dbt, so this bootstrap DDL still lives here.
-#
-# The Semantic (L2) and Marts (L3) layers are managed by dbt. Use `dbt run`, not this script.
-# See gd_dbt/ and docs/03_OPERATIONS.md.
-#
-# SAFETY: files whose header carries a DO NOT RUN or NOT THE LIVE SHAPE banner are skipped, and
-# -Force deliberately does not override that. Two files in warehouse/L1 are CREATE OR REPLACE
-# against tables holding 2.6 million rows of production data.
+# Applies one explicitly allowlisted L1 migration to gooddollar.BlockchainEvents.
+# It never scans the SQL directory. Unknown and historical files are refused by filename.
+# Default execution is plan-only. Production execution requires both switches and a typed prompt.
+# See docs/03_OPERATIONS.md for the migration and approval requirements.
 #
 # Usage:
-#   .\scripts\deploy-warehouse.ps1        # applies the L1 DDL that is safe to re-apply
+#   .\scripts\deploy-warehouse.ps1 -Migration 09_CreateRawLogs_v1.sql
+#   .\scripts\deploy-warehouse.ps1 -Migration 09_CreateRawLogs_v1.sql -Execute -AllowProduction
 #
 # Requires:
 #   - Google Cloud SDK installed (provides the `bq` CLI)
 #   - `gcloud auth application-default login` already run
+#   - Production execution is only for an administrator after separate approval
 
 param(
-    [Parameter(Position = 0)]
-    [ValidateSet("L1")]
-    [string]$Layer = "L1",
+    [Parameter(Mandatory = $true)]
+    [string]$Migration,
+
+    [switch]$Execute,
 
-    [switch]$Force
+    [switch]$AllowProduction,
+
+    [string]$ImpersonateServiceAccount
 )
 
+function New-BqProcessStartInfo {
+    param(
+        [string]$BqExe,
+        [string[]]$Arguments,
+        [string]$ImpersonateServiceAccount
+    )
+
+    $startInfo = New-Object System.Diagnostics.ProcessStartInfo
+    $startInfo.FileName = $env:ComSpec
+    $startInfo.Arguments = '/d /s /c ""' + $BqExe + '" ' + ($Arguments -join ' ') + '"'
+    $startInfo.UseShellExecute = $false
+    $startInfo.CreateNoWindow = $true
+    $startInfo.RedirectStandardInput = $true
+    $startInfo.RedirectStandardOutput = $true
+    $startInfo.RedirectStandardError = $true
+    $startInfo.EnvironmentVariables['CLOUDSDK_AUTH_IMPERSONATE_SERVICE_ACCOUNT'] = $ImpersonateServiceAccount
+    return $startInfo
+}
+
 $ErrorActionPreference = "Stop"
 $ScriptDir = Split-Path -Parent $MyInvocation.MyCommand.Path
 $RepoRoot = Split-Path -Parent $ScriptDir
 $WarehouseDir = Join-Path $RepoRoot "warehouse"
+$MigrationDir = Join-Path $WarehouseDir "L1"
+$AllowedMigrations = @(
+    "08_PipelineRunsOutcome_v1.sql",
+    "09_CreateRawLogs_v1.sql",
+    "10_AddOracleReconciliationCompatibility_v1.sql",
+    "11_CreateRawLogsAllHistory_v1.sql",
+    "12_CreateTransactionsAllHistory_v1.sql"
+)
+
+if ($Migration -notin $AllowedMigrations) {
+    throw "REFUSED_UNLISTED_MIGRATION: '$Migration' is not in the deployment allowlist. Historical and unknown SQL is never executed by this helper."
+}
+
+$MigrationPath = Join-Path $MigrationDir $Migration
+if (-not (Test-Path -LiteralPath $MigrationPath -PathType Leaf)) {
+    throw "Allowlisted migration is missing: $MigrationPath"
+}
+
+$Sql = Get-Content -LiteralPath $MigrationPath -Raw
+if (-not $Sql.Contains('${PROJECT}') -or -not $Sql.Contains('${DATASET}')) {
+    throw "Migration must use the literal project and dataset placeholders: $Migration"
+}
+$Sql = $Sql.Replace('${PROJECT}', 'gooddollar').Replace('${DATASET}', 'BlockchainEvents')
+if ($Sql -match '\$\{(PROJECT|DATASET)\}') {
+    throw "Unresolved identifier placeholder in $Migration"
+}
+
+Write-Host "Migration: $Migration"
+Write-Host "Target: gooddollar.BlockchainEvents"
+
+if (-not $Execute) {
+    Write-Host "PLAN ONLY. No BigQuery client was resolved and no query was submitted."
+    return
+}
+
+if (-not $AllowProduction) {
+    throw "REFUSED_PRODUCTION_EXECUTION: production execution requires -AllowProduction after separate production authorization."
+}
+
+if ($ImpersonateServiceAccount -notmatch '^[A-Za-z0-9._+-]+@[A-Za-z0-9.-]+\.iam\.gserviceaccount\.com$') {
+    throw "REFUSED_IMPERSONATION_REQUIRED: provide the separately approved commissioner service account with -ImpersonateServiceAccount."
+}
+
+$ExpectedConfirmation = "APPLY APPROVED DDL TO gooddollar.BlockchainEvents"
+$Confirmation = Read-Host "Type '$ExpectedConfirmation' to continue"
+if ($Confirmation -cne $ExpectedConfirmation) {
+    throw "Production confirmation did not match. Nothing was submitted."
+}
 
 # Resolve the bq.cmd location (gcloud SDK ships it as bq.cmd on Windows).
 # Try common paths; fall back to PATH lookup.
@@ -49,71 +115,27 @@ if (-not $BqExe) {
     exit 1
 }
 Write-Host "Using bq: $BqExe"
-
-function Invoke-SqlFile {
-    param([string]$Path)
-    Write-Host ""
-    Write-Host "==== $Path ====" -ForegroundColor Cyan
-    $sql = Get-Content -Raw -Path $Path
-    # bq query reads SQL from stdin
-    $sql | & $BqExe query --use_legacy_sql=false --format=none --project_id=gooddollar
-    if ($LASTEXITCODE -ne 0) {
-        Write-Error "bq query failed for $Path"
-        exit $LASTEXITCODE
-    }
-}
-
-function Deploy-Layer {
-    param([string]$LayerName)
-    $layerDir = Join-Path $WarehouseDir $LayerName
-    if (-not (Test-Path $layerDir)) {
-        Write-Error "Layer folder not found: $layerDir"
-        exit 1
-    }
-    $files = Get-ChildItem -Path $layerDir -Filter "*.sql" | Sort-Object Name
-    if ($files.Count -eq 0) {
-        Write-Warning "No .sql files in $layerDir"
-        return
-    }
-
-    # Refuse anything that would drop a table holding production data. warehouse/L1 now contains
-    # historical DDL that is NOT the live shape: 01 and 02 are CREATE OR REPLACE against the two
-    # tables holding 2.6 million rows, and 03 was superseded. Running this folder end to end used
-    # to be safe and no longer is. Each such file carries a banner and is skipped by name.
-    $skipped = @()
-    $toRun = @()
-    foreach ($f in $files) {
-        $head = Get-Content -Path $f.FullName -TotalCount 20 -Raw
-        if ($head -match 'DO NOT RUN|NOT THE LIVE SHAPE') {
-            $skipped += $f.Name
-        } else {
-            $toRun += $f
-        }
-    }
-
-    if ($skipped.Count -gt 0) {
-        Write-Host ""
-        Write-Host "SKIPPED (superseded or destructive, banner in file header):" -ForegroundColor Yellow
-        foreach ($s in $skipped) { Write-Host "  $s" -ForegroundColor Yellow }
-        if ($Force) {
-            Write-Error "-Force does not override this. These files would drop tables holding production data. Run the individual statements you actually want, by hand."
-            exit 1
-        }
-    }
-
-    if ($toRun.Count -eq 0) {
-        Write-Warning "Nothing to run in $LayerName after skips."
-        return
-    }
-
-    Write-Host "Deploying $($toRun.Count) file(s) in $LayerName..." -ForegroundColor Green
-    foreach ($f in $toRun) {
-        Invoke-SqlFile -Path $f.FullName
+$bqProcess = New-Object System.Diagnostics.Process
+$bqProcess.StartInfo = New-BqProcessStartInfo -BqExe $BqExe -ImpersonateServiceAccount $ImpersonateServiceAccount -Arguments @(
+    'query', '--use_legacy_sql=false', '--format=none', '--project_id=gooddollar', '--maximum_bytes_billed=10737418240'
+)
+try {
+    Write-Host "Executing $Migration as $ImpersonateServiceAccount" -ForegroundColor Cyan
+    [void]$bqProcess.Start()
+    $stdoutTask = $bqProcess.StandardOutput.ReadToEndAsync()
+    $stderrTask = $bqProcess.StandardError.ReadToEndAsync()
+    $bqProcess.StandardInput.WriteLine($Sql)
+    $bqProcess.StandardInput.Close()
+    $bqProcess.WaitForExit()
+    $stdout = $stdoutTask.Result
+    $stderr = $stderrTask.Result
+    if ($stdout) { Write-Host $stdout }
+    if ($bqProcess.ExitCode -ne 0) {
+        throw "bq query failed for $Migration with exit code $($bqProcess.ExitCode): $stderr"
     }
-    Write-Host "$LayerName complete." -ForegroundColor Green
+    if ($stderr) { Write-Host $stderr }
+} finally {
+    $bqProcess.Dispose()
 }
 
-Deploy-Layer $Layer
-
-Write-Host ""
-Write-Host "Done." -ForegroundColor Green
+Write-Host "Migration completed." -ForegroundColor Green