feat(node): graceful shutdown + Prometheus metrics endpoint (#22)

* feat(db): versioned schema migrations with idempotent backfill

Replace the inline &str SQL array that re-runs on every node startup with a
versioned migration system. Each migration is recorded in a new
schema_migrations table and applied at most once per node, inside a single
transaction.

For nodes upgrading from a pre-migration-versioning build, the canonical
`repos` table is used as a signal to mark v1 as already applied without
re-running its ~140 statements — so existing operators see zero behavior
change on first restart after this commit.

Closes the gap called out in docs/OSS-READINESS-AUDIT.md:78.

- Adds Db::migration_status() returning applied (version, name, applied_at)
- Adds 6 unit tests validating the static migration catalogue
  (versions strictly increasing, names distinct, bodies non-empty, v1
  name locked to initial_schema)
- cargo fmt + clippy -D warnings + cargo test --workspace all clean

* fix(db): make legacy migration backfill safe and lock-guarded

Address two robustness gaps in the v1 migration runner on the live network:

- Remove the "repos table present => v1 complete" backfill short-circuit.
  It assumed every existing node already had the full current schema; a node
  that was behind would be marked v1-applied while still missing newer tables
  or columns. Every v1 statement is idempotent (CREATE TABLE/INDEX IF NOT
  EXISTS, ADD COLUMN IF NOT EXISTS), so legacy installs now simply run v1 once:
  existing objects are no-ops, missing ones get created.

- Guard the whole runner with a Postgres advisory lock so two instances
  against the same database (blue/green or rolling deploy) can't race to apply
  the same migration and trip the schema_migrations primary key. The lock is
  released explicitly, and automatically if the connection drops.

Also document that per-migration transactions preclude CREATE INDEX
CONCURRENTLY in future migrations.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(node): graceful shutdown + Prometheus metrics endpoint

Two operational-safety changes that the live network needs:

1) Graceful shutdown. A process-wide tokio::sync::watch::Sender<bool>
   in AppState is flipped on SIGINT (all platforms) or SIGTERM (Unix).
   Every long-lived loop now selects on the shutdown arm:
     * axum server (with_graceful_shutdown — drains in-flight)
     * libp2p swarm (drops Swarm, closes connections)
     * gossip task (exits between peer announces)
     * sync worker (exits between batches)
     * operator heartbeat loop
     * rate-limit cleanup loop
     * peer-count metrics poller
     * the deferred DID-record publish task

   New GITLAWB_SHUTDOWN_GRACE_SECS (default 30s) controls how long
   axum waits for in-flight requests before returning.

2) Prometheus /metrics endpoint. New metrics module exposes:
     * gitlawb_info{version, did}                 (constant gauge = 1)
     * gitlawb_pushes_total{repo}                 (counter)
     * gitlawb_fetches_total{repo}                (counter)
     * gitlawb_auth_successes_total{route}        (counter, helper)
     * gitlawb_auth_failures_total{route,reason}  (counter, helper)
     * gitlawb_sync_queue_processed_total{status} (counter)
     * gitlawb_webhook_deliveries_total{result}   (counter)
     * gitlawb_pack_size_bytes                    (histogram)
     * gitlawb_peers_connected                    (gauge)

   Opt-in via GITLAWB_METRICS_ADDR (e.g. 127.0.0.1:9091). The endpoint
   is on a separate listener so the public port stays unexposed;
   bind to localhost or a private interface.

3) Metric instrumentation hooks in api/repos.rs (push/fetch),
   webhooks.rs (delivery outcomes), and sync.rs (per-item status).
   The auth helpers are defined but not yet wired — that's a
   focused follow-up.

Tests:
  * 3 new metrics::tests::* — encode returns valid prometheus text,
    record_* helpers are no-ops before init(), encode() errors if
    init() was never called.
  * cargo fmt + clippy -D warnings + cargo test --workspace all clean.

Closes the metrics and graceful-shutdown gaps in
docs/OSS-READINESS-AUDIT.md:131 and docs/MAINTAINER-ROADMAP.md:35.

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

Author: Vasanthdev2004

Cargo.lock

@@ -51,6 +51,10 @@ aws-config = { version = "1", features = ["behavior-version-latest"] }
 async-compression = { version = "0.4", features = ["tokio", "zstd"] }
 tar = "0.4"
 zstd = "0.13"
+# Prometheus metrics. Used to expose a /metrics endpoint for ops/observability
+# on the opt-in GITLAWB_METRICS_ADDR listener. The crate is also the de-facto
+# exposition format encoder in the Rust ecosystem.
+prometheus = { version = "0.13", default-features = false }
 alloy = { version = "1", default-features = false, features = [
     "contract",
     "provider-http",

crates/gitlawb-node/Cargo.toml

@@ -366,7 +366,8 @@ pub async fn git_upload_pack(
         .acquire(&record.owner_did, &record.name)
         .await
         .map_err(|e| AppError::Git(e.to_string()))?;
-    smart_http::upload_pack(&disk_path, body)
+    let body_len = body.len();
+    let resp = smart_http::upload_pack(&disk_path, body)
         .await
         .map_err(|e| {
             let msg = e.to_string();
@@ -377,7 +378,10 @@ pub async fn git_upload_pack(
                 tracing::error!(repo = %name, err = %msg, "git-upload-pack failed");
                 AppError::Git(msg)
             }
-        })
+        })?;
+    crate::metrics::record_fetch(&format!("{owner}/{name}"));
+    crate::metrics::observe_pack_size(body_len as f64);
+    Ok(resp)
 }
 
 /// POST /:owner/:repo.git/git-receive-pack  (AUTH REQUIRED — enforced by middleware)
@@ -451,6 +455,7 @@ pub async fn git_receive_pack(
         })?;
     let disk_path = guard.path().to_path_buf();
     tracing::debug!(repo = %name, path = %disk_path.display(), "running git receive-pack");
+    let body_len = body.len();
     let receive_result = smart_http::receive_pack(&disk_path, body).await;
 
     // Always release the advisory lock — even on error — to prevent stale locks
@@ -465,6 +470,11 @@ pub async fn git_receive_pack(
     // Update the repo's updated_at timestamp after a successful push
     let _ = state.db.touch_repo(&record.id).await;
 
+    // Record the successful push for metrics. The body has already been
+    // consumed by smart_http::receive_pack so we observe size up front.
+    crate::metrics::record_push(&record.id);
+    crate::metrics::observe_pack_size(body_len as f64);
+
     // Record push event for trust score and issue a signed ref certificate
     let pusher_did = extract_did_from_auth(&headers);
     if let Some(ref did) = pusher_did {

crates/gitlawb-node/src/api/repos.rs

@@ -502,6 +502,7 @@ mod tests {
             machine_id: None,
             repo_store: crate::git::repo_store::RepoStore::for_testing(PathBuf::from("/tmp"), pool),
             rate_limiter: RateLimiter::new(100, Duration::from_secs(60)),
+            shutdown_tx: tokio::sync::watch::channel(false).0,
         }
     }
 

crates/gitlawb-node/src/auth/mod.rs

@@ -131,6 +131,19 @@ pub struct Config {
     /// Default: 2 GB.  Set lower on resource-constrained nodes.
     #[arg(long, env = "GITLAWB_MAX_PACK_BYTES", default_value_t = 2_147_483_648)]
     pub max_pack_bytes: usize,
+
+    /// Optional address to bind a Prometheus `/metrics` exposition endpoint on.
+    /// Example: `127.0.0.1:9091`. Leave empty (default) to disable.
+    /// Bind to localhost or a private interface — the metrics endpoint is
+    /// unauthenticated.
+    #[arg(long, env = "GITLAWB_METRICS_ADDR", default_value = "")]
+    pub metrics_addr: String,
+
+    /// Maximum time to wait for in-flight requests to drain on shutdown, in
+    /// seconds. After this elapses, the server returns 503 to anything still
+    /// in flight and exits. Default: 30s.
+    #[arg(long, env = "GITLAWB_SHUTDOWN_GRACE_SECS", default_value_t = 30)]
+    pub shutdown_grace_secs: u64,
 }
 
 impl Config {