ArgoCD root app for the vmubtkube-a cluster. Everything in this repo is
applied by the vmubtkube-a Application (application.vmubtkube-a.yaml),
which prunes and self-heals via ServerSideApply.
Upstream manifests that don't ship as a Helm chart — operators (CNPG,
cert-manager, contour), plugin bundles (barman-cloud), standalone CRDs
(external-secrets) — are vendored with
vendir instead of committed as ad-hoc
YAML. vendir fetches a whole upstream release atomically — manifest, CRDs,
and RBAC together — so an image bump can't silently leave CRDs stale (which is
what happened before this setup). Only the tag/ref in vendir.yml is
edited; everything else is generated.
vendir.yml # sources + pinned tags/refs (the only hand-edited file)
vendir.lock.yml # fetched content digests (generated)
vendored/<name>/
base/ # upstream files, written by `vendir sync` — never hand-edit
kustomization.yaml # present only when local patches are needed
- Refresh: bump a
tag/refinvendir.yml, runvendir sync, then commitvendir.yml,vendir.lock.yml, and the changedvendored/*/base/together. - Renovate bumps
vendir.ymlvia itsvendirmanager and runsvendir sync(postUpgradeTasks) so the refreshed base files land in the same PR — no manual re-download. - Local changes (private-registry image mirrors, resource limits, config) go
in a kustomize overlay next to the base. Never edit
base/— the next sync overwrites it. - ArgoCD renders each vendored component from its own Application: a plain
directory source when there's no overlay, or a kustomize source when a
kustomization.yamlis present. Base files are committed, so ArgoCD never runs vendir at sync time.
Manifests that do ship as Helm charts stay as ArgoCD Helm-source apps
(application.<name>.yaml); image-only deployments remain hand-written and are
tracked by Renovate's kubernetes manager. Full design:
docs/superpowers/specs/2026-07-01-vendir-kustomize-design.md.
CI runs .ci/validate.sh (kubeconform, strict; overlays validated as
kustomize build output) on every PR. To run the same check locally on each
commit:
pre-commit install # needs kubeconform, pluto, and kustomize-or-kubectl on PATH
Schemas are cached under .tmp/, so repeat runs take ~2s.
CI itself caches two things across runs in a dedicated rustfs bucket
(ci-cache) in the cluster, restored and saved by the cache-restore /
cache-save steps:
charts/<chart>-<version>.tgz— Helm chart tarballs (no more 23index.yamlround trips per pipeline, ~90s)schemas/<repo>.git.tar— git object databases for the kubeconform schema mirrors. The 734 MB checkout is rebuilt locally from the cached sparse-checkout config, but the cached objects are not small: measured 144 MB forkubernetes-json-schema+ 16 MB forCRDs-catalog, i.e. ~160 MB moved per pipeline (the bucket's ownrustfs_bucket_usage_usage_bytesagrees at 168 MB total). That buys ~12 s (13.7 s cold clone → 1.3 s warm) and removes a GitHub dependency from the critical path, so it is kept deliberately — it is cluster-local traffic, not internet traffic
The validation binaries stay plain wgets: caching them saved ~1.5s while
moving ~90 MB per pipeline into the bucket, so it was measured and dropped. The
secrets the cache steps use are rows in Woodpecker's own database, kept in sync
with the rustfs-ci-cache Secret by woodpecker/cache-secrets.woodpecker.yaml
— rotating the Secret needs no manual step.
Cache contents are fail-open: both steps run with failure: ignore, saves are
additive, and a miss only costs time. The cache credential is not: Woodpecker
resolves from_secret while compiling the pipeline, so if cache_access_key /
cache_secret_key are missing the whole pipeline fails before any step runs.
Those rows are maintained by woodpecker/cache-secrets.woodpecker.yaml (a
CronJob that upserts them from the rustfs-ci-cache Secret), which therefore has
to be deployed and to have run once — treat it as infrastructure, not a
convenience. woodpecker/rustfs.woodpecker.yaml
deploys the endpoint; design in
docs/superpowers/specs/2026-11-06-ci-cache-rustfs-design.md.
Two DNS zones, split by audience:
-
*.apps.somemissing.info— internet-facing services, exposed through the Contour edge (HTTPProxy) with public DNS. -
*.k8s.somemissing.info— internal/LAN-only services, reachable from the home network but not from the internet. These resolve to routable ServiceIPs via theexternal-dns.alpha.kubernetes.io/hostnameannotation on a plain ClusterIP Service (external-dns writes thek8s.somemissing.infozone over RFC2136).Why a bare ClusterIP is reachable from off-cluster: Calico BGP-peers with the default gateway (
172.16.3.254, AS 64512 on both sides) and advertises the Service CIDR192.168.208.0/20to it — see theBGPConfiguration/BGPPeer/BGPFilterresources inapplication.calico.yaml. The gateway therefore routes ClusterIPs straight to the nodes, so LAN clients connect to the ServiceIP directly. No LoadBalancer, NodePort, or Ingress is needed to expose a service on the LAN — reach for one only when you actually want L7 features, and prefer the*.appsContour edge for those. (A separate192.168.240.0/28pool is advertised for the few LoadBalancer Services that do exist, e.g. syslog ingest, which needsexternalTrafficPolicy: Localto preserve the sender's source IP.)
Both zones get publicly-trusted Let's Encrypt certificates from the
cert-manager-webhook-dnsimple-production ClusterIssuer. DNS-01 validation
has no reachability requirement, so internal-only *.k8s hostnames are
issued the same way as public ones — no private CA to trust on clients. See
grafana.yaml for a minimal example, or proxy_argocd_webhook.yaml for a
cert consumed directly by the workload (argocd-server-tls).
The private k8s CA in clusterissuer.yaml is not for browser-facing
certs — it issues internal service-to-service certs (e.g.
python-envoy-authz, validated by Contour via the delegated k8s-ca
secret).
Two mechanisms are available; pick by where the secret originates, not by what kind of thing it is:
Credentials that exist solely to satisfy an in-cluster service — database
passwords, app admin passwords, inter-service tokens that nothing outside the
cluster ever needs — should be auto-generated in the cluster by the
mittwald kubernetes-secret-generator
(installed in operators, see application.secret-generator.yaml).
Declare the Secret empty in git with the trigger annotation; the controller
fills data.password with a random string on first sync:
apiVersion: v1
kind: Secret
metadata:
name: example-password
namespace: default
annotations:
secret-generator.mittwald.de/secret-type: password
secret-generator.mittwald.de/password-length: "32"
type: OpaqueWhy prefer this:
- Nothing to pre-create or rotate by hand — the value is generated once and persisted in-cluster.
- No secret material in git, and no dependency on the external store for bootstrapping.
- Under ServerSideApply the controller owns the
datafield, so ArgoCD self-heal won't clobber it. (Rotate by deleting/regenerating the Secret; the consuming workload restarts via Reloader.)
See mumble.yaml for a working example.
External Secrets Operator (installed via
application.external-secrets.yaml) syncing from the Bitwarden-backed
ClusterSecretStore named default is primarily for secrets whose value is
defined off-platform — third-party API tokens, upstream service
credentials, pre-existing accounts whose password can't be invented here. The
source of truth is Bitwarden, and the cluster reflects it.
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: example-config
namespace: default
spec:
refreshInterval: 300s
secretStoreRef:
name: default
kind: ClusterSecretStore
target:
name: example-config
data:
- secretKey: api-token
remoteRef:
key: example-api-tokenSee selfoss.yaml for a working example.
Occasionally a secret is intentionally checked into git because there is no
sensible off-platform source and no in-cluster generator (e.g. an opaque key
the receiver doesn't even enforce). That should be rare and always accompanied
by a note in docs/ explaining why — see docs/hyperdx-api-key.md.