feat(extensions): add StaticKeysTenantExtension — env-configured per-user API keys with per-schema isolation

[rafaelkallis]

Summary

Adds a TenantExtension mapping static API keys (from env) to per-user PostgreSQL schemas — shipped in the extensions registry (hindsight-extensions/static-keys-tenant/), not the server, per review: it follows the supabase-tenant layout (package, test-only pyproject.toml, Dockerfile, README, tests, CI job). Closes #3674.

Fully self-hosted multi-user memory isolation with no users table and no new services: a bridge between ApiKeyTenantExtension (single shared key) and SupabaseTenantExtension (external IdP).

Configuration

HINDSIGHT_API_TENANT_EXTENSION=hindsight_ext_static_keys_tenant:StaticKeysTenantExtension
HINDSIGHT_API_TENANT_USERS=user1:key1,user1:key2,user2:key3   # required; multiple keys may map to one user
HINDSIGHT_API_TENANT_SCHEMA_PREFIX=user                       # optional, default "user" → user_<user_id> schemas

HINDSIGHT_API_TENANT_MCP_AUTH_DISABLED is deliberately not supported — setting it fails at startup (MCP clients always authenticate with a user's key; no isolation bypass).

Behavior

• Authorization: Bearer <key> → authenticated as the mapped user
• Per-user schema {prefix}_{user_id} provisioned lazily on first access, serialized per schema with an asyncio.Lock (no concurrent double-migration); provisioning failure raises AuthenticationError (not cached)
• User ids normalized (lowercased, dashes → underscores) so case variants collapse onto one canonical schema, consistent with Postgres identifier folding; collisions (dash normalization, >63-byte identifiers) rejected at startup
• Keys must be ASCII and comma-free (documented); constant-time comparison on bytes via hmac.compare_digest
• Unknown/missing key → 401 (AuthenticationError); non-ASCII bearer tokens → 401, never a 500
• authenticate() sets context.tenant_id and a stable non-secret api_key_id (truncated sha256 of the key) — metering can distinguish a user's multiple keys
• Config error messages never echo key material (entry index / user id / derived key_id only)
• list_tenants() returns all configured users so the worker and maintenance sweep per-user schemas; hindsight-admin run-db-migration pre-provisions them (documented)
• /health stays public

Testing

• 49 unit tests (uv run pytest in the package, no DB): init validation (missing/invalid config, SQL-injection ids, schema collisions, duplicate keys, key-format edge cases), auth (valid/invalid/missing/non-ASCII, metering fields, per-key ids, schema provision-once + concurrency + failure-not-cached), MCP (auth required, flag refused), list_tenants, loader integration, package entrypoint
• CI: test-extension-static-keys-tenant runs the suite and builds the image on the latest-slim base (detect-changes filter + gate wired)
• ruff / ruff format --check / ty clean

Compatibility

Fully opt-in; no changes to the server package. Only activates when HINDSIGHT_API_TENANT_EXTENSION points at it.

Related

• Closes Feature Request: StaticKeysTenantExtension — env-configured per-user API keys with per-schema isolation #3674 (feature request)
• Ref Feature Request: Team, agent, and memory-level access control #2235 (authorization discussion — deliberately small env-only slice), Feature: per-request LLM credentials for multi-user / enterprise deployments #2932 (per-request LLM credentials — this provides the per-identity anchor)

[Sanderhoff-alt]

The authentication path does not behave as the description states: the constant-time fallback can never match, and it turns a bad key into a 500 for some inputs. Two other points on tenant-id collisions and the untyped key map.

[rafaelkallis]

Fixing a tenant-isolation edge case found in review: mixed-case user IDs

During code review we found that StaticKeysTenantExtension allowed mixed-case user IDs (e.g. Rafael), which produced mixed-case schema names (user_Rafael). That is unsafe:

• The migration path creates the schema quoted (CREATE SCHEMA IF NOT EXISTS "user_Rafael"), preserving case.
• But the runtime's fq_table() does not quote the schema (f"{schema}.{table}"), so Postgres folds it to lowercase.
• Result: either every query for such a user fails ("schema does not exist"), or — worse — two users like Rafael and rafael silently collapse onto one schema, breaking the extension's core isolation guarantee.

Fix (commit 4f0f820e): normalize user IDs to lowercase at __init__ before building the schema name. Case variants (Rafael / rafael / RAFAEL) now resolve to a single canonical tenant and schema, consistent with Postgres identifier folding. Distinct users whose ids collide in non-case ways (e.g. jane-doe vs jane_doe, or ids colliding past the 63-byte identifier limit) are still rejected loudly, as before.

Added regression tests covering: lowercase normalization, mixed-case + dash normalization, case-insensitive duplicate-user acceptance, and mixed-case dash-collision rejection, plus an auth-level test asserting a mixed-case key resolves onto the lowercased schema and a list_tenants() test for the lowercased tenant id.

[nicoloboschi]

please move this work to https://github.com/vectorize-io/hindsight/tree/main/hindsight-extensions --- thanks!

[rafaelkallis]

Done — the work has been moved to the extensions registry as requested.

What changed (PR now shows only these files):

• New standalone package: hindsight-extensions/static-keys-tenant/ with hindsight_ext_static_keys_tenant/ (package), pyproject.toml (test harness only), Dockerfile (image packaging), README.md, and tests/ — mirroring supabase-tenant.
• Registry entry added in hindsight-extensions/README.md.
• Developer docs updated: hindsight-docs/docs/developer/extensions.md now points at the registry.

Nothing remains in hindsight-api-slim — the builtin multi_key_tenant.py and its exports are gone, since the extension is no longer bundled with the server (consistent with the supabase-tenant precedent).

The extension is imported via HINDSIGHT_API_TENANT_EXTENSION=hindsight_ext_static_keys_tenant:StaticKeysTenantExtension, has no third-party dependencies beyond the server, and its 40 unit tests pass (uv run pytest, no DB needed — matching the registry dev workflow).

[rafaelkallis]

@nicoloboschi thanks for the pointer to the registry — the move is fine and the package now follows the supabase-tenant layout. Just want to put the reasoning behind this extension on the record, since I would argue it sits on the "broadly useful" side of the slim boundary:

• Zero third-party dependencies. It imports only stdlib plus the server's own hindsight_api.* interfaces — the Dockerfile has no pip install step, and it runs on the latest-slim base unchanged. It would add nothing to the slim image.
• Primary motivation: one user, many harnesses. A single-user deployment can mint multiple access tokens for the same user so each agent harness (Claude Code, Codex, Cursor, …) authenticates with its own key while sharing one memory schema — with per-key attribution in usage metering. That is a common self-hosted setup, not a niche one.
• Secondary: self-hosted multi-user isolation. Per-profile schema isolation without an IdP or users table — e.g. running multiple Hermes profiles, each with its own isolated memory banks under its own key, all on one deployment. Exactly the gap between the shared-key builtin and the vendor-bound Supabase extension.

If you would prefer this bundled in hindsight-all-slim after all, say the word — it is a minimal move on my side. Otherwise I am happy to merge as is, in the registry.

[nicoloboschi]

Thanks — the move to hindsight-extensions/ is done properly: the layout, pyproject.toml, Dockerfile (with ARG HINDSIGHT_IMAGE), entrypoint test, registry row and docs blurb all mirror supabase-tenant. I checked the branch out locally: 40 tests pass, ruff check and ruff format --check are clean.

Two things must change before merge, plus a few smaller ones.

Must fix

1. No CI job — none of this ever runs in CI.
supabase-tenant has test-extension-supabase-tenant (.github/workflows/test.yml:3928), a detect-changes filter (:180) and a gate entry (:5603), and that job also builds its image. This PR touches no workflow file, so its 40 tests and its Dockerfile are never exercised on any future change. Please add the same three pieces.

2. API keys leak into startup logs. See inline comments — the plaintext key ends up in ValueError messages on misconfiguration.

Should fix

3. Undocumented key-format constraints. A key cannot contain a comma (it is the entry separator), and a non-ASCII key can never authenticate: header values arrive latin-1-decoded while env values are utf-8-decoded, so encode("utf-8", "surrogateescape") on the two sides does not produce the same bytes. It fails closed, which is fine, but a permanent silent 401 is a miserable debugging session — please state "keys must be ASCII and must not contain a comma" in the README.

4. mcp_auth_disabled on a multi-user extension. You raised this yourself; my vote is refuse at init rather than ship parity. On ApiKeyTenantExtension the flag downgrades one shared key to none. Here it hands anyone unauthenticated MCP access to the base schema, in a deployment whose entire purpose is per-user isolation.

Nits

• list_tenants() returning every configured user rather than only the provisioned ones is the right call for run-db-migration, and better than what supabase-tenant does. It does mean the poller's fallback path issues one EXISTS probe per configured user on every idle cycle, against schemas that may not exist yet (worker/poller.py:440; the exception is swallowed, so it is harmless). Worth a README line pointing at hindsight-admin run-db-migration to pre-provision.
• Two concurrent first requests from the same user both call run_migration. Inherited from supabase-tenant, so not a blocker, but a per-schema asyncio.Lock would close it.
• The branch is behind main (still merges cleanly), and the PR description is stale: it still describes hindsight_api.extensions.builtin.multi_key_tenant and "30 tests".

[rafaelkallis]

All points addressed — one commit per item, pushed to the PR branch:

Must fix #1 — CI job (ec27d176): test-extension-static-keys-tenant mirrors the supabase wiring — detect-changes filter + output mapping for hindsight-extensions/static-keys-tenant/**, a job that runs uv sync + pytest tests -v and builds the image on the latest-slim base, and the job added to the report-pr-status gate.

Must fix #2 — key leaks (1b11e1a1 + inline replies): config errors now report the entry index / validated user id / derived key_id — never entry or the key. Regression tests assert the messages contain no key material.

Should fix #3 — key constraints documented (57bc49d0): README states keys must be ASCII and comma-free, with the latin-1/utf-8 explanation for why a non-ASCII key would otherwise fail closed with a permanent silent 401. Same commit documents hindsight-admin run-db-migration for pre-provisioning (the poller EXISTS-probe nit).

Should fix #4 — mcp_auth_disabled (a64e31a7): adopting your vote — the variable now raises ValueError at init (documented in the variable table), and authenticate_mcp always delegates to authenticate(). The parity behavior is gone.

Nits:

• Concurrent first provision per schema now serialized with a per-schema asyncio.Lock + re-check (f112daf9).
• Configured keys are pre-encoded to compare_digest-ready bytes at init; per-request encoding is only the incoming key (6fe2a989), and the constant-time comment now states honestly that the loop stops at the first match.

PR description rewritten (the old one still said builtin.multi_key_tenant and "30 tests").

Checks: 50 tests pass (uv run pytest), ruff check / ruff format --check / ty check all clean.

Branch is rebased onto current main tip (7e17a120) and merges cleanly. Re-requested your review.