Skip to content

feat(chart): image normaliser config and egress NetworkPolicy #74

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
pjb157 wants to merge 5 commits into
base: main
Choose a base branch
from
Draft

feat(chart): image normaliser config and egress NetworkPolicy #74

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
3b64e6e
feat(chart): egress NetworkPolicy and Workload Identity wiring
pjb157 May 19, 2026
3695623
docs(values): GCS bucket lifecycle guidance for image normaliser
pjb157 May 20, 2026
20ab664
fix(chart): default clusterCidrs to empty and clarify when to set it
pjb157 May 20, 2026
241289e
docs(values): note on non-standard DNS pod labels for NetworkPolicy
pjb157 May 21, 2026
9f8d7fd
docs(values): warn that egress NetworkPolicy is incompatible with in-…
pjb157 May 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 65 additions & 0 deletions templates/networkpolicy.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
{{- if .Values.networkPolicy.enabled -}}
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: {{ include "control-layer.fullname" . }}-egress
labels:
{{- include "control-layer.labels" . | nindent 4 }}
spec:
podSelector:
matchLabels:
{{- include "control-layer.selectorLabels" . | nindent 6 }}
app.kubernetes.io/component: control-layer
policyTypes:
- Egress
egress:
# Egress to the public internet, with private / loopback / link-local
# / CGNAT / IPv6 ULA + LL ranges excluded. This is the application-layer
# backstop for the in-process image fetcher's IP allow-list: even if a
# bug let a request through the deny-list inside the process, the CNI
# refuses the packet.
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
# RFC1918 private ranges
- 10.0.0.0/8
- 172.16.0.0/12
- 192.168.0.0/16
# Loopback
- 127.0.0.0/8
# Link-local — covers GCE / AWS metadata at 169.254.169.254

@doubleword-code doubleword-code Bot May 21, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: This comment is misleading and the rule breaks Workload Identity on GKE Dataplane V2.

Why it matters: The 169.254.0.0/16 exception denies egress to this range (it's in the except list under 0.0.0.0/0). However, GKE Dataplane V2 requires pods using Workload Identity to reach the metadata server at 169.254.169.254 on ports 80/8080 for token exchange. The GKE documentation explicitly states:

For clusters running GKE Dataplane V2, allow egress to 169.254.169.254/32 on ports 80 and 8080.

Blocking this IP prevents the image normaliser from obtaining GCP service account credentials, breaking GCS writes entirely.

Suggested fix: Either (a) add a separate allow rule for the metadata server before the broad deny, or (b) carve out 169.254.169.254/32 from the link-local exception:

# Allow GKE metadata server for Workload Identity
- to:
    - ipBlock:
        cidr: 169.254.169.254/32
  ports:
    - protocol: TCP
      port: 80
    - protocol: TCP
      port: 8080

Alternatively, make this rule conditional on a new allowGcpMetadata value that defaults to true for GKE compatibility.

- 169.254.0.0/16

@doubleword-code doubleword-code Bot May 21, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: This CIDR exception denies egress to the GKE metadata server required for Workload Identity.

Why it matters: Combined with line 31's comment, this rule blocks 169.254.169.254/32 which is the GKE metadata server endpoint. Per GKE documentation, Workload Identity on Dataplane V2 clusters requires egress to this IP on ports 80/8080. Without it, the pod cannot obtain GCP service account tokens, breaking GCS object writes from the image normaliser.

Suggested fix: Narrow this exception to exclude the metadata server IP, or add a separate allow rule for 169.254.169.254/32 on ports 80/8080 before this deny rule takes effect.

# Carrier-grade NAT (RFC 6598)
- 100.64.0.0/10
# In-cluster Pod / Service CIDRs (operator-configured)
{{- range .Values.networkPolicy.clusterCidrs }}

@doubleword-code doubleword-code Bot May 22, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Minor YAML indentation inconsistency — the ranged CIDR entries (line 37) are indented with 14 spaces, matching the static RFC1918 entries above, which is correct. However, the closing {{- end }} on line 38 has no trailing content after it, so the next line (39) starts a new - ipBlock: entry. This is fine, but worth double-checking that the rendered YAML has proper structure.

The template looks correct as-is; this is just a note to verify the output once with helm template . before merging.

- {{ . }}
{{- end }}
- ipBlock:

@doubleword-code doubleword-code Bot May 20, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: IPv6 cluster CIDRs are not excluded, defeating the security purpose for IPv6 traffic.

Why it matters: The IPv4 rule (lines 22-38) excludes both standard private ranges AND operator-configured clusterCidrs to prevent in-cluster egress. However, the IPv6 rule (lines 39-47) only excludes loopback (::1/128), ULA (fc00::/7), and link-local (fe80::/10). If your cluster uses IPv6 Pod or Service CIDRs (e.g., fd00::/108 or similar), those would be allowed by the ::/0 rule because they're not in the except block. This creates an asymmetric security posture where IPv4 in-cluster egress is blocked but IPv6 in-cluster egress is permitted.

Suggested fix: Add a mechanism to exclude IPv6 cluster CIDRs. Options:

  1. Add a separate networkPolicy.clusterCidrsIPv6 value and range over it in the IPv6 except block
  2. Support mixed IPv4/IPv6 entries in clusterCidrs and use Helm logic to split them into the appropriate except blocks
  3. Document that IPv6 clusters require manual modification of this template

cidr: ::/0
except:
# IPv6 loopback
- ::1/128
# IPv6 unique-local (RFC 4193)
- fc00::/7
# IPv6 link-local
- fe80::/10
{{- if .Values.networkPolicy.allowKubeDns }}
# DNS is required for any outbound resolution. Without this rule, the
# cluster-CIDR deny above blocks queries to kube-dns and breaks all
# name lookups inside the pod.
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system

@doubleword-code doubleword-code Bot May 20, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: Namespace metadata label requires Kubernetes 1.21+.

Why it matters: The kubernetes.io/metadata.name label on namespaces was added in Kubernetes 1.21. Clusters running older versions won't have this label, causing the DNS allow rule to fail to match and breaking name resolution inside the pod. Additionally, some distributions may use different labels for their DNS service (not k8s-app: kube-dns).

Suggested fix: Either:

  1. Add a note in the documentation specifying Kubernetes 1.21+ requirement for this feature
  2. Use a more compatible namespace selector (e.g., match on namespace name directly if supported by CNI)
  3. Add a configurable label override for clusters with non-standard DNS deployments

@doubleword-code doubleword-code Bot May 22, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: The kube-dns selector uses kubernetes.io/metadata.name: kube-system which requires Kubernetes v1.16+ (when this label was auto-added to all namespaces). For clusters running older versions, this rule won't match and DNS will break.

Why it matters: The comment mentions OpenShift and managed offerings using different labels, but doesn't note the minimum K8s version requirement. Most production clusters are well past v1.16, but it's worth noting for users on legacy distributions.

Suggested fix: Add a brief note in the comment above line 48 mentioning the v1.16+ requirement, or consider using the older namespaceSelector with matchNames: ["kube-system"] if backward compatibility is needed.

podSelector:
matchLabels:

@doubleword-code doubleword-code Bot May 20, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: The kube-dns selector uses k8s-app: kube-dns, which matches the standard CoreDNS deployment in many clusters. However, some managed Kubernetes providers use different labels:

  • EKS: k8s-app: kube-dns (correct) but also adds eks.amazonaws.com/component: kube-dns
  • RKE2: Uses k8s-app: coredns (lowercase)
  • NodeLocal DNSCache: Creates a DaemonSet with different selectors

Why it matters: If an operator enables this NetworkPolicy on a cluster where the DNS pods don't match k8s-app: kube-dns, all outbound DNS queries will fail, breaking name resolution for the control-layer pod.

Suggested fix: Consider either:

  1. Adding a note in the values.yaml documentation to check kubectl get pods -n kube-system --show-labels and adjust the template if needed
  2. Or making the DNS selector configurable via .Values.networkPolicy.dnsSelector for clusters with non-standard DNS deployments

Alternatively, document this as a known requirement: "Operators must verify their cluster's DNS pod labels match k8s-app: kube-dns or modify the template accordingly."

k8s-app: kube-dns

@doubleword-code doubleword-code Bot May 20, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: DNS pod label assumption may not match all clusters.

Why it matters: While k8s-app: kube-dns is common, some Kubernetes distributions use different labels for CoreDNS. For example, some use k8s-app: coredns or other variations. If the label doesn't match, DNS queries will be blocked even with allowKubeDns: true.

Suggested fix: Consider making the DNS pod selector configurable via values, or document that operators may need to adjust this label based on their cluster's DNS deployment.

@doubleword-code doubleword-code Bot May 20, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: This NetworkPolicy assumes your cluster's DNS pods use the label k8s-app: kube-dns in the kube-system namespace, which is correct for vanilla Kubernetes and GKE with default CoreDNS. However, some distributions differ:

Why it matters: If your cluster uses a different DNS deployment (e.g., OpenShift uses dns.operator.openshift.io/daemonset-dns=default in openshift-dns namespace), enabling this NetworkPolicy without adjustment will break in-pod DNS resolution, preventing all outbound traffic that requires name lookup.

Suggested fix: Add a note in the values.yaml comment block (near line 380) alerting operators to verify their DNS pod labels before enabling, or consider making the DNS selector configurable:

# In values.yaml
dnsSelector:
  namespace: kube-system
  podLabels:
    k8s-app: kube-dns

Then template it dynamically. For now, at minimum document this assumption for operators.

ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
{{- end }}
{{- end }}
54 changes: 53 additions & 1 deletion values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,23 @@ serviceAccount:
create: true
# Automatically mount a ServiceAccount's API credentials?
automount: true
# Annotations to add to the service account
# Annotations to add to the service account.
#
# For GKE Workload Identity (used when the image normaliser writes
# objects to a GCS bucket), bind a GCP service account here:
#
# annotations:
# iam.gke.io/gcp-service-account: <name>@<project>.iam.gserviceaccount.com
#
# The bound GCP service account needs `roles/storage.objectAdmin` on the
# image-normaliser bucket and `roles/iam.serviceAccountTokenCreator` on
# itself (to call signBlob for V4 signed URLs).
#
# Bucket lifetime: dwctl does NOT garbage-collect normaliser objects.
# Configure an Object Lifecycle Management rule on the bucket itself
# (Terraform / gcloud) to delete objects after enough days to outlive
# the longest possible batch dispatch window. Example: 7d (covers a
# 24h batch completion window + a 6d investigation buffer).
annotations: {}
# The name of the service account to use.
# If not set and create is true, a name is generated using the fullname template
Expand Down Expand Up @@ -328,3 +344,39 @@ postgresql:
runAsGroup: 999
# Security best practice - prevent running as root
runAsNonRoot: true

# ---------------------------------------------------------------------------
# Egress NetworkPolicy for the control-layer pod.
#
# Disabled by default. When enabled, restricts the control-layer pod's
# outbound traffic so it cannot reach private / loopback / link-local
# ranges or in-cluster service CIDRs. Public-internet egress (provider
# APIs, external Postgres, object storage) is preserved.
#
# This is the application-layer backstop for the hardened image fetcher:
# even if a bug in the in-process IP allow-list let a request through,
# the kernel-level network policy refuses the packet.
#
# The template ALREADY denies RFC1918 (10.0.0.0/8, 172.16/12, 192.168/16),
# loopback, link-local, CGNAT, and IPv6 ULA/LL ranges. `clusterCidrs`
# is only needed if your cluster uses Pod or Service CIDRs OUTSIDE
# those ranges (e.g. dual-stack with public IPv6, custom CIDR plans).
#
# For standard GKE clusters (default Pod CIDR 10.x and Service CIDR
# 10.x), the hardcoded RFC1918 block already covers them — leave the
# default empty list and nothing additional is denied.
#
# Misconfiguring this can break in-pod DNS resolution if `kube-dns`
# isn't allowed; the template emits an explicit `kube-dns` allow rule
# regardless of which CIDRs you set.
#
# Requires a NetworkPolicy-aware CNI (Cilium, Calico, GKE Dataplane V2).
networkPolicy:
enabled: false
# Additional CIDRs to deny egress to, ON TOP OF the hardcoded RFC1918
# / loopback / link-local / CGNAT / IPv6-ULA-LL exceptions. Empty by
# default — standard clusters using RFC1918 Pod/Service ranges need
# nothing here. Set explicitly only for non-RFC1918 cluster CIDRs.
clusterCidrs: []
# Allow egress to kube-dns. Disable only if you know what you're doing.
allowKubeDns: true
Loading