Feature/key custody

[keltschdt]

Pull Request

This is stage 2 in a family of drafts. Stage 1 is contained in #16448

Summary

When OPENID_REUSE_TOKENS is on and a user logs in with OpenID, the server today keeps the IdP token set in four places it can read back: req.session.openidTokens, the browser refreshToken cookie, a refreshtokenbridges document, and the single-flight result in openidrefreshflights. Every one is plaintext or decryptable with CREDS_KEY, which the process already has. So anyone who reads a database or backup dump together with the process environment gets working IdP refresh tokens for every session in that dump. That is a bulk leak that needs no live access.

It happens under normal use: OPENID_REUSE_TOKENS on, an OpenID login, and later read access to a dump.

After this change the server keeps no readable copy of the token set. On login it mints a 32-byte key, sends it to the browser only in the httpOnly openid_token_key cookie, and seals the token set with AES-256-GCM under that key into one tokencustodies record keyed by sha256(key) (tokenKeyHash). It also signs a marker cookie openid_user_id holding { id, tokenKeyHash }. The key is never written to any record, session, flight, or log, so the stored blob cannot be opened with the database or CREDS_KEY alone. The cookie value stays the same across token rotation, so a rotation re-seals the existing record instead of issuing a new key. The refresh-token bridge is no longer needed, and is removed. TOTP secrets move under a separate TOTP_KEY that defaults to CREDS_KEY (this separation allows for a future split-container setup where an auth-only instance must not have access to CREDS_KEY).

Because the server no longer keeps a readable token set, every feature that forwarded the user's IdP tokens is moved onto the same custody record, obtained lazily. The OpenID placeholders ({{LIBRECHAT_OPENID_ACCESS_TOKEN}}, _ID_TOKEN, _TOKEN, _USER_ID, _USER_EMAIL, _USER_NAME, _EXPIRES_AT), the Microsoft Graph placeholder ({{LIBRECHAT_GRAPH_ACCESS_TOKEN}}), and the Graph token endpoint (GET /api/auth/graph-token, the SharePoint picker) resolve the live token set through the per-request custody loader instead of req.user.federatedTokens. A request opens custody only when it actually resolves a token-bearing placeholder, exchanges a Graph token, or runs OBO / direct bearer, and all of those share the one memoized custody read per request; an ordinary message or conversation request opens custody never. This closed a regression the custody migration would otherwise introduce: a custody-login browser session carries the id_token as its Bearer, so its federatedTokens snapshot is empty, and without this change those placeholders would raise OpenIDReauthRequiredError or resolve empty and the Graph endpoint would answer 401.

How it works

Login mints the key and writes the sealed record. Refresh rotates the sealed record under the same key. File authorization proves the browser holds the key without ever opening the blob. The per-request loader reads the store at most once, and file authorization reads only projected metadata, never the sealed bytes.

sequenceDiagram
    participant Browser
    participant API
    participant Store as tokencustodies
    participant IdP
    Browser->>API: OpenID login (setOpenIDAuthTokens)
    API->>API: generateTokenKey + hashTokenKey
    API->>Store: createCustody -> sealed record (rotationCounter 0)
    API->>Browser: Set-Cookie openid_token_key + openid_user_id
    Browser->>API: POST /api/auth/refresh
    API->>API: loadOpenIDCustody -> openCustody (one read, memoized)
    API->>IdP: refresh grant (only when the token is not fresh)
    IdP->>API: new token set
    API->>Store: rotateCustody -> compare-and-set on rotationCounter
    API->>Browser: re-issue openid_token_key (same value, new expiry)
    Browser->>API: GET /images/:ownerId/:filename
    API->>API: verifyCustodyBinding -> findTokenCustodyMeta (blob untouched)
    API->>Browser: authorized file

Loading

The behavior lives in packages/api and packages/data-schemas. The /api CJS files keep only requires, route registration, and calls into those modules.

packages/data-schemas/src/schema/tokenCustody.ts   # TokenCustody schema/model + store methods
packages/api/src/auth/custody/
├── key.ts        # key generate/parse/hash + openid_token_key cookie helpers
├── aead.ts       # sealTokens / openTokens (AES-256-GCM)
├── service.ts    # createCustody / openCustody / rotateCustody / reloadAfterInvalidGrant / custodyExists / delete
├── loader.ts     # per-request, memoized loadOpenIDCustody
├── binding.ts    # verifyCustodyBinding for file and shared-link authorization
└── graph.ts      # resolveGraphApiToken for GET /api/auth/graph-token (SharePoint)
packages/api/src/mcp/openid.ts                      # resolveOpenIDPlaceholderTokens (lazy)
packages/api/src/utils/{oidc,env,graph}.ts          # placeholder resolvers take resolved tokens
packages/api/src/auth/totp/secret.ts               # TOTP_KEY resolution (defaults to CREDS_KEY)
config/migrate-key-custody.js                       # migration + index install + legacy purge

The change also removes the RefreshTokenBridge store and its modules, config, and drift detection; the OpenID sessions row writes; and the plaintext single-flight result (the openidrefreshflights entry is now sealed under the session key).

Three behavior changes land alongside the token-source move. The custody record supplies the tenant: reads address the record by tokenKeyHash alone (the unique index identifies one record), and cross-tenant protection is a comparison of the record's own tenantId after the read against the tenant the caller knows, so the unauthenticated /api/auth/refresh — which knows no tenant up front — opens a tenant-stamped record and runs under the tenant it reads back; this fixes tenant users being forced to re-login on every refresh. A refresh whose record is deleted by a concurrent logout mid-rotation fails closed: rotateCustody reports gone (compare-and-set matched nothing and the reload found no live record) rather than reviving the pre-rotation session, the refresh is rejected session-missing with no cookie re-issued, and the refresh token the IdP just granted — which the rotation never persisted — gets one best-effort IdP revocation so it is not stranded live. The publication-generation response-delivery guard (withOpenIDResponseDelivery and the flight delivery methods / publicationFlight* fields) is removed, because the custody record is now the liveness check: refresh proves liveness through the compare-and-set, and the Graph endpoint rechecks custodyExists immediately before sending.

Lazy versus eager. The lazy design above resolves tokens only on requests that forward them, keeping ordinary requests off the custody store. The alternative — a middleware on every MCP-reaching route that opens custody and fills req.user.federatedTokens, leaving all MCP and Graph readers untouched — was not taken: it opens custody (one indexed read plus one AES-GCM open) on every such request whether or not any token is used, which covers most chat traffic and counts against the Lighthouse query budget, and its snapshot does not inline-refresh, so a long stream can hold an expired token. Eager is only viable with a per-runner, in-memory cache of the sealed record (ciphertext plus metadata, keyed by tokenKeyHash, short TTL, opened per request with the cookie's key so no plaintext outlives the request); that cache's cost is that a logout, ban or deletion on one runner is not seen by another runner's cache until its TTL expires. The same sealed-record cache is the deferred option for the lazy path if the one read per token-forwarding request ever proves to matter.

Type of change

• Breaking change
• Tests / tooling / CI

Testing

npx tsc --noEmit is clean in packages/api and packages/data-schemas.

Tests pass:

• packages/api custody/openid/utils/mcp: 45 suites, 829 tests (AEAD round-trip, wrong key, associated data, hash binding, nondeterminism, tamper; service lifetime/counter/stability/non-destructive; loader and binding; the gone rotation outcome and orphan-token revocation; resolveOpenIDPlaceholderTokens including Property 19 — custody is opened only for token-bearing placeholders — as an exhaustive placeholder × placement × source matrix; processMCPEnv/extractOpenIDTokenInfo resolved-token precedence; the Graph placeholder precedence; OBO delegation).
• packages/data-schemas tokenCustody and openidRefreshFlight: 33 tests (real unique and TTL indexes, the expiresAt > now read, the atomic compare-and-set under concurrency, lookup by hash alone with the tenant returned on the record, findTokenCustodyMeta projecting userId/tenantId, the by-user delete tenant filter both ways, index install with MONGO_AUTO_INDEX=false, flight behavior with the delivery methods removed).
• api CJS auth suites: 167 tests across AuthController (refresh custody path, tenant-stamped refresh, record-deleted-mid-rotation, graph controller onto custody with the custodyExists recheck), AuthController.custody.route, LogoutController, OpenIDRefreshFlight, openIdJwtStrategy (empty federatedTokens snapshot, legacy session/cookie reads removed), and banViolation.

npm run sort-imports applied to the touched paths only.

npm run lighthouse (the LCP budget lane: in-memory Mongo, 250 ms added per query) passed locally, run in a clean worktree at this head with no .env and an en-US browser locale: 1 passed, medians LCP 3840.92 ms / 4500, CLS 0.0169 / 0.1, TBT 52.8 ms / 500. The unmodified base under identical conditions measured LCP 3971.608 ms, CLS 0.0169, TBT 48.2 ms, so the change sits within run-to-run noise. Locally the lane failed in global setup at the "Sign up" link whenever a repo-root .env disabled email login or the OS locale was not English; both reproduced on the base too. This branch pins the en-US locale wherever a browser starts outside Playwright Test's own en-US default: the setup pages in e2e/setup/authenticate.ts and e2e/setup/record.js, and the Chrome the Lighthouse CLI launches in e2e/lighthouse/audit.ts (--accept-lang=en-US, because --lang is ignored on macOS). A .env that disables email login is overridden by exporting ALLOW_EMAIL_LOGIN=true ALLOW_REGISTRATION=true. With these in place, npm run lighthouse:run passes from a checkout with its .env present (LCP 3787.5 ms, CLS 0.0169, TBT 59.5 ms). The loader reads the store at most once per request, file authorization is one projected findTokenCustodyMeta read that never opens the blob, refresh reuses the loaded context, and no frontend or message-loading code changed.

Risk / compatibility

One-time re-login. There is no read-compatibility path: every active OpenID session logs in once more at rollout. A request with only an old refreshToken cookie, or a session holding openidTokens, or a marker with only a refreshTokenHash claim gets 401 { code: 'OPENID_SESSION_MISSING' }.

No user-facing change beyond that re-login, which shows the normal login screen.

Record lifetime is not a new setting. It is derived per token response: max(refreshTokenExpiresAt, accessTokenExpiresAt), or max(receipt time + REFRESH_TOKEN_EXPIRY, accessTokenExpiresAt) when the response gives no refresh-token expiry, recomputed on every rotation. REFRESH_TOKEN_EXPIRY is only the fallback term; its local-auth role (the sessions row expiry and the local refreshToken cookie lifetime) is unchanged.

OPENID_REFRESH_BRIDGE_GRACE_MS is removed with the bridge. The server behaves the same whether it was set or not.

Token forwarding, tenant, and the Graph endpoint

MCP servers and the SharePoint picker that forward the user's IdP tokens keep working, now sourced from the custody record. A chat request that drives an MCP server configured with an obo: block, an Authorization: Bearer {{LIBRECHAT_OPENID_ACCESS_TOKEN}} header, or any {{LIBRECHAT_OPENID_*}} / {{LIBRECHAT_GRAPH_ACCESS_TOKEN}} placeholder now performs one indexed custody read (shared across all of them for that request); a request that forwards no IdP token performs none, and nothing is added to the page-startup path. On the base, every request paid a session-store read for the same data, so single-tenant and non-MCP traffic is flat or lighter.

Multi-tenant deployments: tenant users can now refresh. The record carries its own tenantId, the refresh looks the record up by key and reads the tenant back, and assertOpenIDSessionIdentityMatch still rejects a cross-tenant record before any IdP call. No new header or claim is required; this matches how the tenant used to come from the server-held session.

GET /api/auth/graph-token behaves as before for a live session, and answers 401 { code: 'OPENID_SESSION_MISSING' } (clearing the OpenID cookies) when no live custody record backs the request, instead of the previous "No federated access token" 401. The remote-agent bearer path, which carries its own verified upstream token and has no custody record, is unchanged.

One deviation worth noting for reviewers: the id-token-preference refresh follow-up is not performed. The upstream provider's tokenPreference is fixed at creation and the MCP call sites build it once with access_token preference; when a config carries both an access-type placeholder and {{LIBRECHAT_OPENID_ID_TOKEN}}, the id token from the access-preference resolution is used as-is, and processOpenIDPlaceholders validates its own expiry and raises OpenIDReauthRequiredError if it is stale — a fail-closed, actionable outcome. Threading a per-preference provider factory through the four MCP layers was judged out of proportion; revisit if a deployment needs the id token specifically force-refreshed.

TOTP_KEY and the migration

Leave TOTP_KEY unset and nothing changes: the key resolves to CREDS_KEY, existing TOTP secrets (v3, v2, and bare) keep verifying, and no migration is needed. The migration's --reencrypt-totp step refuses to run in this case, because there is nothing to move.

Setting TOTP_KEY to a new value is what needs the migration. Existing secrets are still sealed under CREDS_KEY, so they must be re-encrypted under the new key with --reencrypt-totp. The step moves only the secrets that open under CREDS_KEY but not under TOTP_KEY; it skips ones already under TOTP_KEY and leaves alone ones that open under neither (re-encrypting those would seal garbage over the real secret). Each moved secret is written as a v3: blob — the format change is a side effect of re-encrypting, not a separate cleanup pass. Plaintext is unchanged, so a user who verified before keeps verifying once the server can read the secret under the new key.

There is an unavoidable transition window, and its ordering does not remove it. Two things have to change: the server's key (CREDS_KEY → TOTP_KEY, which needs a restart) and the secrets at rest (re-encrypted by the migration, which takes time). A secret verifies only when the server's key matches the secret's key, so every affected user fails 2FA during the gap between those two changes — whichever you do first:

• Migrate first: secrets are on TOTP_KEY while the servers still read CREDS_KEY, so everyone is broken until you restart the servers.
• Restart first: servers read TOTP_KEY while the secrets are still on CREDS_KEY, so everyone is broken until the migration finishes.

Either way the first action breaks affected users and the second heals them; both are required. This code resolves a single key with no fallback, so it cannot try TOTP_KEY then CREDS_KEY, which is the only thing that would close the window. Plan the two actions close together and expect 2FA failures in between. Keep the window short.

One safety point about the migration itself: --reencrypt-totp must be able to read the current secrets to move them, so run it with CREDS_KEY still available. If it cannot read a secret under the resolved key it treats it as "opens under neither key" and skips it, which would silently strand that secret.

Access tokens in the custody

With only the database, a custody record cannot be opened: sealedTokens is AES-256-GCM under the browser's key, and the record holds just sha256(key). Given the openid_token_key cookie value of one session (for debugging that session), this mongosh script finds the record and decrypts it. It uses the same steps as key.ts and aead.ts: hash the 32 raw key bytes, look up by tokenKeyHash, then open the kc1:iv:tag:ciphertext blob with the record's identity as associated data.

const crypto = require('crypto');
const tokenKey = 'PASTE_TOKEN_KEY_FROM_BROWSER_HERE';

// 1. tokenKey -> tokenKeyHash
const key = Buffer.from(tokenKey, 'base64url');
const tokenKeyHash = crypto.createHash('sha256').update(key).digest('base64url');
print('tokenKeyHash: ' + tokenKeyHash);

// 2. tokenKeyHash -> custody record
const rec = db.tokencustodies.findOne({ tokenKeyHash });
if (!rec) { print('no custody record (logged out, expired or a different login)'); quit(1); }

// 3. Decrypt sealedTokens (kc1:iv:tag:ciphertext, AES-256-GCM, identity bound as AAD)
const [fmt, iv, tag, body] = rec.sealedTokens.split(':');
const aad = Buffer.from(JSON.stringify([
  'kc1', tokenKeyHash, rec.userId,
  rec.tenantId ?? null, rec.openidIssuer ?? null, rec.openidSubject ?? null,
]), 'utf8');
const d = crypto.createDecipheriv('aes-256-gcm', key, Buffer.from(iv, 'base64url'));
d.setAAD(aad);
d.setAuthTag(Buffer.from(tag, 'base64url'));
const payload = JSON.parse(
  Buffer.concat([d.update(Buffer.from(body, 'base64url')), d.final()]).toString('utf8'),
);
printjson(payload); // prints the live tokens in full

Run it as a file (mongosh LibreChat --file custody-decrypt.js). A long --eval or a multi-line paste into the interactive shell can mangle it. The output is accessToken, idToken, refreshToken, accessTokenExpiresAt, refreshTokenExpiresAt and issuedAt; refreshTokenExpiresAt is absent when the IdP sends no refresh_expires_in (Entra ID, for example). A wrong key or an identity field that differs from the sealed one fails with Unsupported state or unable to authenticate data, and GCM returns no partial plaintext. The printed refresh token is live, so treat the output like a password; print payload.refreshToken.length to check without exposing it.

Accepted residual risks (by design, not fixed here)

1. ADMIN_OAUTH_EXCHANGE holds a plaintext IdP refresh token for 30 seconds, single use, in the admin exchange-code cache. Closing it is a separate change.
2. A live server compromise still sees every token for any session that makes a request during the compromise. Custody narrows the attack from retroactive-and-bulk to live-and-per-session.
3. MCP and action OAuth tokens stay decryptable under CREDS_KEY, by scope decision.
4. TOTP secrets stay decryptable under TOTP_KEY (default CREDS_KEY): the verifying request is not authenticated yet, so no user-held key is available.
5. Every active OpenID session logs in once more at rollout.
6. A user who clears cookies or switches browsers loses the session. The orphaned record is inert and TTL reclaims it.
7. Cookie headers in logs or at a proxy are now key material. The openid_token_key cookie is no more exposed than the refreshToken cookie it replaces, but do not log cookie headers.
8. A TOTP_KEY change has the transition window described above; a single-key resolver cannot avoid it.

Checklist

• I reviewed my own changes
• Relevant tests have been added or updated
• Existing relevant tests pass
• The change does not introduce new warnings or errors
• User-facing or complex behavior is documented where necessary
• Required dependency changes have been merged/published — N/A, no dependency added
• Required documentation PR: N/A

---

Joachim Keltsch on behalf of Daimler Truck AG.
Provider & Legal Notice | Daimler Truck
Copyright (c) 2025 Daimler Truck AG

[github-actions]

👋 Thanks for the contribution! LibreChat merges all changes into dev first — main only moves at release time — so this pull request's base branch was switched from main to dev automatically.

Nothing is needed from you; your commits, reviews and discussion are unchanged. If the diff now shows files you did not touch, rebase onto dev:

git remote add upstream https://github.com/LibreChat-AI/LibreChat.git
git fetch upstream dev
git rebase upstream/dev
git push --force-with-lease

Maintainers: apply the target: main label and restore the base branch if this one genuinely belongs on main.