Session invalidation when scaling Kratos replicas in Kubernetes

[treylade]

Session invalidation when scaling Kratos replicas in Kubernetes

Environment

• Ory Kratos version: v1.3.1
• Kubernetes deployment using official Helm chart (version 0.50.7)
• Database: PostgreSQL
• Service Mesh: Linkerd
• CNI: Flannel
• External Access: Cloudflare Tunnels (no K8s Ingress)
• Deployment Scale: 3 replicas

Description

When scaling Ory Kratos from 1 to multiple replicas in a Kubernetes environment, sessions become invalid intermittently. While the root cause is not yet determined, one hypothesis is that it might be related to session cache inconsistency between replicas, though this is purely speculative and needs validation.

Steps to Reproduce

1. Deploy Ory Kratos using the official Helm chart
2. Configure custom NGINX proxy for HTTPS termination
3. Scale deployment to multiple replicas
4. Authenticate user through frontend application
5. Make subsequent API calls using the obtained session

Current Behavior

• With 1 replica: Sessions work perfectly
• With >1 replicas: Random "No valid session found" errors occur during:
  • Direct Frontend-to-Kratos via Cloudflare Tunnel and NGINX Proxy obtained ory_kratos_session (no issue)
  • Subsequent internal Service-to-Service communication via internal DSN of Kratos Public / Admin Service API forwarding the initially obtained ory_kratos_session (no valid session found error comes up)

Expected Behavior

Sessions should remain valid regardless of which replica handles the request.

Configuration

# Relevant parts of values.yaml
replicaCount: 3

deployment:
  annotations:
    linkerd.io/inject: enabled

kratos:
  config:
    serve:
      public:
        base_url: https://auth.example.org/
        cors:
          enabled: true
          allowed_origins:
            - https://app.example.org
          allowed_methods:
            - POST
            - GET
            - PUT
            - PATCH
            - DELETE
          allowed_headers:
            - Authorization
            - Cookie
            - Content-Type
          allow_credentials: true

    session:
      cookie:
        persistent: true
      lifespan: 24h

Service architecture:

Frontend -> Cloudflare Tunnel -> NGINX Proxy -> Kratos Public API (internal port 80)
Quart Microservices -> Internal cluster DNS -> Kratos Public & Admin APIs
- Public API: For session validation, user data retrieval
- Admin API: For privileged operations (internal only)

Additional Context

1. All services (including NGINX proxies) are replicated 3 times
2. Session is stored in PostgreSQL database
3. While all data is properly stored in PostgreSQL, the exact cause of session invalidation is unclear. The in-memory session caching hypothesis is just one possible explanation that needs investigation
4. No Kubernetes Ingress is used and shall not be used; all external traffic routes through Cloudflare Tunnels (thus configuring Ingress via Ory Kratos values.yaml is not an option)

Attempted Solutions

1. Implemented session affinity at the Service level:

sessionAffinity: ClientIP
sessionAffinityConfig:
  clientIP:
    timeoutSeconds: 10800  # 3 hours

2. Ensured proper pod distribution using topology spread constraints and anti-affinity rules

Questions

Has anyone successfully run Ory Kratos with multiple replicas in a Kubernetes environment? We're experiencing session invalidation issues when scaling beyond a single replica, and would greatly appreciate any insights or recommendations from the community regarding horizontal scaling of Kratos.

Relevant Links

• Similar issue discussion
• Official K8s deployment documentation

Config files / Logs

Please advise what further config files and logs would be helpful, and I'll provide them.

[treylade]

Closed: Moved to ory/k8s#733