Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 39 additions & 1 deletion kustomize/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,7 @@ Additional features that can be enabled:

### Optional but Recommended

- **Ingress Controller** – Traefik is the recommended default. Install it yourself (e.g. below or via Helm). Optional overlay components `ingress-nginx` and `aws-load-balancer-controller` only patch the plane Ingress with controller-specific annotations. See [AWS_load_balancer_setup.md](AWS_load_balancer_setup.md) for the ALB controller.
- **Ingress Controller** – Traefik is the recommended default. Install it yourself (e.g. below or via Helm). Optional overlay components `ingress-nginx` and `aws-load-balancer-controller` only patch the plane Ingress with controller-specific annotations. See [AWS_load_balancer_setup.md](AWS_load_balancer_setup.md) for the ALB controller. On OpenShift use `ingress-openshift` instead, which swaps the Ingress for native Routes — see [Running on OpenShift](#running-on-openshift).
```bash
# Add Traefik Helm repo and install Traefik ingress controller
helm repo add traefik https://traefik.github.io/charts
Expand Down Expand Up @@ -620,6 +620,44 @@ components:

No extra configuration is required. The component patches all base Deployments (api, web, space, admin, live, worker, etc.) with the same non-root security context. Ensure your container images support running as the configured user (e.g. UID 1000); Plane Commercial images are built to run as non-root.

## Running on OpenShift

OpenShift is the inverse case: it refuses to let you pick the UID at all. The default `restricted-v2` SCC ignores the image's `USER`, assigns an arbitrary UID from the namespace's range, and places the process in group 0. It also validates the pod's own request with `MustRunAsRange`, so a manifest asking for a *specific* `runAsUser`/`runAsGroup`/`fsGroup` outside that range is **rejected at admission** — meaning `nonroot-security-context`, which pins 1000, stops every pod from scheduling.

**Start from [`overlays/openshift/`](overlays/openshift/README.md)**, which wires this up and ships its own `vars.yaml.example` / `secrets-vars.yaml.example`:

```bash
cd overlays/openshift
cp vars.yaml.example vars.yaml
cp secrets-vars.yaml.example secrets-vars.yaml
# edit both, then:
oc new-project plane-openshift
oc kustomize . | oc apply -f -
```

Copying `default` and toggling components does not work — it pins `runAsUser: 1000` and ships the in-cluster datastores, and OpenShift rejects both. The overlay differs from `default` in exactly three ways:

```yaml
# overlays/openshift/kustomization.yaml
components:
- ../../components/openshift-security-context # NOT nonroot-security-context
- ../../components/ingress-openshift # NOT ingress-nginx / ingress-traefik
# postgres / redis / rabbitmq / minio / opensearch omitted — external only
```

| Component | What it does |
| --------- | ------------ |
| `openshift-security-context` | Same hardening as `nonroot-security-context` — `runAsNonRoot`, `capabilities.drop: [ALL]`, `seccompProfile: RuntimeDefault` — but names no UID, so the SCC assigns one. |
| `ingress-openshift` | Removes the base nginx `Ingress` and adds one `route.openshift.io/v1` Route per path, each with `haproxy.router.openshift.io/timeout: 300s`. Carries its own `APP_DOMAIN` replacement, so no extra overlay wiring is needed. |

Three things to know:

- **Image requirement.** The images must grant group 0 write access to the paths they write at runtime. Older images crash under an arbitrary UID — nginx exits with `mkdir() "/var/cache/nginx/client_temp" failed (13: Permission denied)` and celery beat cannot create its schedule DB.
- **Remove the bundled datastores.** `postgres`, `redis`, `rabbitmq`, `minio` and `opensearch` are third-party images with baked-in UID and data-directory ownership. They cannot run under an arbitrary UID, and (as with `nonroot-security-context`) the security-context component does not patch them — they are listed after it in the overlay. Drop those components, point the overlay at managed services, or grant their ServiceAccounts a relaxed SCC.
- **Upgrading an existing deployment is safe.** Moving a running install from the pinned-uid-1000 posture to this one needs no data migration: kubelet re-applies `fsGroup` to PVC contents on mount, so data written by the old deployment stays readable and writable by the new UID.

The alternative to `ingress-openshift` is to keep the base `Ingress` and set `INGRESS_CLASS: "openshift-default"` in `vars.yaml`, letting OpenShift's ingress-to-route controller do the conversion. That works, but the controller only converts an `Ingress` whose class maps to `openshift.io/ingress-to-route` (`nginx` is ignored), and whether the per-route HAProxy annotations survive the conversion varies by OCP version. Note that **Routes have no request-body size limit** — Traefik's `plane-body-limit` middleware has no equivalent, so enforce upload limits in the application or at a WAF/CDN.

## Configuration

### Configuration System
Expand Down
64 changes: 64 additions & 0 deletions kustomize/components/ingress-openshift/kustomization.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component

# OpenShift ingress: replaces the base nginx Ingress with one
# route.openshift.io/v1 Route per path, served by the cluster's HAProxy router.
#
# Pair with the `openshift-security-context` component; do NOT also enable
# `ingress-nginx`, `ingress-traefik` or `aws-load-balancer-controller`.
#
# WHY EXPLICIT ROUTES
#
# OpenShift's ingress-to-route controller can convert a plain Ingress into Routes,
# and that path also works: skip this component, keep the base Ingress, and set
# INGRESS_CLASS to "openshift-default" in vars.yaml (the controller only picks up
# an Ingress whose class maps to openshift.io/ingress-to-route -- "nginx" is
# ignored). But Plane needs a per-route HAProxy timeout, and whether annotations
# survive that conversion varies by OCP version. Declaring the Routes removes the
# guesswork.
#
# WHAT IS LOST vs THE TRAEFIK PATH
#
# - No body-size limit. Traefik's `plane-body-limit` Middleware
# (buffering.maxRequestBodyBytes) has no Route equivalent -- HAProxy cannot cap
# request bodies per route. Enforce upload limits in the app or at a WAF/CDN.
# - No path rewriting. A path needing a prefix strip (e.g. the draw-io component's
# stripPrefix) requires `haproxy.router.openshift.io/rewrite-target` on that
# Route instead, and Routes cannot express Traefik's redirectRegex at all.
# - Path-based Routes require edge or reencrypt termination; passthrough cannot
# do path routing.
#
# TLS: these Routes edge-terminate on the Ingress Operator's wildcard certificate.
# To serve your own cert, add `spec.tls.externalCertificate.name: <tls-secret>`
# (OpenShift 4.16+, and the router needs a RoleBinding to read that Secret).

resources:
- routes.yaml

# The base nginx Ingress is inert on OpenShift (no controller serves class
# "nginx", and ingress-to-route ignores it). Remove it so the Routes are the only
# ingress surface and there is nothing stale to mislead an operator.
patches:
- target:
kind: Ingress
name: plane-ingress
patch: |
$patch: delete
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: plane-ingress

# The overlay's own replacement targets `kind: Ingress` for the host, which no
# longer exists once the patch above runs. Carry the equivalent here so an overlay
# needs no extra wiring beyond enabling this component.
replacements:
- source:
kind: ConfigMap
name: overlay-vars
fieldPath: data.APP_DOMAIN
targets:
- select:
kind: Route
fieldPaths:
- spec.host
164 changes: 164 additions & 0 deletions kustomize/components/ingress-openshift/routes.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,164 @@
# OpenShift Routes — one per path, mirroring base/ingress.yaml.
#
# `host` is the placeholder that the overlay's DOMAIN_NAME/APP_DOMAIN replacement
# rewrites (see kustomization.yaml in this directory, which carries the matching
# replacement so the overlay needs no extra wiring).
#
# Ordering is irrelevant: HAProxy matches the most specific path prefix, so the
# "/" catch-all cannot shadow the others the way it would in Traefik.
apiVersion: route.openshift.io/v1
kind: Route
metadata:
name: plane-route-spaces
namespace: plane
annotations:
# The router default is 30s, which severs /live/'s collaborative-editing
# WebSockets and /pi/'s streaming responses. Applied uniformly for simplicity.
haproxy.router.openshift.io/timeout: 300s
spec:
host: plane.example.com
path: /spaces/
to: { kind: Service, name: plane-space, weight: 100 }
port: { targetPort: 3000 }
tls: { termination: edge, insecureEdgeTerminationPolicy: Redirect }
---
apiVersion: route.openshift.io/v1
kind: Route
metadata:
name: plane-route-god-mode
namespace: plane
annotations:
haproxy.router.openshift.io/timeout: 300s
spec:
host: plane.example.com
path: /god-mode/
to: { kind: Service, name: plane-admin, weight: 100 }
port: { targetPort: 3000 }
tls: { termination: edge, insecureEdgeTerminationPolicy: Redirect }
---
apiVersion: route.openshift.io/v1
kind: Route
metadata:
name: plane-route-api
namespace: plane
annotations:
haproxy.router.openshift.io/timeout: 300s
spec:
host: plane.example.com
path: /api/
to: { kind: Service, name: plane-api, weight: 100 }
port: { targetPort: 8000 }
tls: { termination: edge, insecureEdgeTerminationPolicy: Redirect }
---
apiVersion: route.openshift.io/v1
kind: Route
metadata:
name: plane-route-auth
namespace: plane
annotations:
haproxy.router.openshift.io/timeout: 300s
spec:
host: plane.example.com
path: /auth/
to: { kind: Service, name: plane-api, weight: 100 }
port: { targetPort: 8000 }
tls: { termination: edge, insecureEdgeTerminationPolicy: Redirect }
---
apiVersion: route.openshift.io/v1
kind: Route
metadata:
name: plane-route-graphql
namespace: plane
annotations:
haproxy.router.openshift.io/timeout: 300s
spec:
host: plane.example.com
path: /graphql/
to: { kind: Service, name: plane-api, weight: 100 }
port: { targetPort: 8000 }
tls: { termination: edge, insecureEdgeTerminationPolicy: Redirect }
---
apiVersion: route.openshift.io/v1
kind: Route
metadata:
name: plane-route-marketplace
namespace: plane
annotations:
haproxy.router.openshift.io/timeout: 300s
spec:
host: plane.example.com
path: /marketplace/
to: { kind: Service, name: plane-api, weight: 100 }
port: { targetPort: 8000 }
tls: { termination: edge, insecureEdgeTerminationPolicy: Redirect }
---
apiVersion: route.openshift.io/v1
kind: Route
metadata:
name: plane-route-dav
namespace: plane
annotations:
haproxy.router.openshift.io/timeout: 300s
spec:
host: plane.example.com
path: /dav/
to: { kind: Service, name: plane-api, weight: 100 }
port: { targetPort: 8000 }
tls: { termination: edge, insecureEdgeTerminationPolicy: Redirect }
---
apiVersion: route.openshift.io/v1
kind: Route
metadata:
name: plane-route-live
namespace: plane
annotations:
haproxy.router.openshift.io/timeout: 300s
spec:
host: plane.example.com
path: /live/
to: { kind: Service, name: plane-live, weight: 100 }
port: { targetPort: 3000 }
tls: { termination: edge, insecureEdgeTerminationPolicy: Redirect }
---
apiVersion: route.openshift.io/v1
kind: Route
metadata:
name: plane-route-silo
namespace: plane
annotations:
haproxy.router.openshift.io/timeout: 300s
spec:
host: plane.example.com
path: /silo/
to: { kind: Service, name: plane-silo, weight: 100 }
port: { targetPort: 3000 }
tls: { termination: edge, insecureEdgeTerminationPolicy: Redirect }
---
apiVersion: route.openshift.io/v1
kind: Route
metadata:
name: plane-route-pi
namespace: plane
annotations:
haproxy.router.openshift.io/timeout: 300s
spec:
host: plane.example.com
path: /pi/
to: { kind: Service, name: plane-pi-api, weight: 100 }
port: { targetPort: 8000 }
tls: { termination: edge, insecureEdgeTerminationPolicy: Redirect }
---
# Catch-all LAST, purely for readability.
apiVersion: route.openshift.io/v1
kind: Route
metadata:
name: plane-route-web
namespace: plane
annotations:
haproxy.router.openshift.io/timeout: 300s
spec:
host: plane.example.com
path: /
to: { kind: Service, name: plane-web, weight: 100 }
port: { targetPort: 3000 }
tls: { termination: edge, insecureEdgeTerminationPolicy: Redirect }
46 changes: 46 additions & 0 deletions kustomize/components/openshift-security-context/kustomization.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component

# OpenShift "restricted-v2" SCC security context.
#
# Use this INSTEAD OF `nonroot-security-context` on OpenShift. The two differ in
# exactly one respect: this one does not name a UID.
#
# OpenShift ignores the image's USER, assigns an arbitrary UID from the
# namespace's range, and places the process in group 0. It also validates the
# pod's own request with MustRunAsRange, so a manifest asking for a specific
# runAsUser / runAsGroup / fsGroup outside that range is REJECTED at admission --
# `nonroot-security-context` pins 1000 and therefore prevents every pod from
# scheduling. Everything else (runAsNonRoot, dropped capabilities, seccomp) is
# kept; only the IDs are left for the platform to assign.
#
# REQUIRES images that grant group 0 write access to the paths they write at
# runtime (plane-ee #9018). Older images crash under an arbitrary UID -- nginx
# exits with `mkdir() "/var/cache/nginx/client_temp" failed (13: Permission
# denied)` and celery beat cannot create its schedule DB.
#
# NOT for the bundled datastores. postgres / redis / rabbitmq / minio /
# opensearch are third-party images with baked-in UID and data-directory
# ownership; they cannot run under an arbitrary UID. On OpenShift use managed
# services and leave those components out of the overlay, or grant their
# ServiceAccounts a relaxed SCC.

patches:
- path: patches/pod-security-context.json
target:
kind: Deployment
- path: patches/container-security-context.json
target:
kind: Deployment
- path: patches/pod-security-context.json
target:
kind: StatefulSet
- path: patches/container-security-context.json
target:
kind: StatefulSet
- path: patches/pod-security-context.json
target:
kind: Job
- path: patches/container-security-context.json
target:
kind: Job
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
[
{
"op": "add",
"path": "/spec/template/spec/containers/0/securityContext",
"value": {
"allowPrivilegeEscalation": false,
"capabilities": {
"drop": ["ALL"]
},
"runAsNonRoot": true,
"seccompProfile": {
"type": "RuntimeDefault"
}
}
}
]
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
[
{
"op": "add",
"path": "/spec/template/spec/securityContext",
"value": {
"runAsNonRoot": true,
"seccompProfile": {
"type": "RuntimeDefault"
}
}
}
]
Loading