Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 10 additions & 5 deletions helm-charts/nullify-k8s-collector/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ The chart refuses to render when a required value is missing or still a placehol
|-----------|-------------|---------|
| `cloudProvider` | Platform the collector runs on: `aws` (EKS) or `gcp` (GKE) | `aws` |
| `serviceAccount.create` | If true, create a new service account | `true` |
| `serviceAccount.requireRoleArn` | EKS: fail the render unless `eks.amazonaws.com/role-arn` is an IAM role ARN (partitions `aws`, `aws-us-gov`, `aws-cn`). Set `false` when the pod gets AWS credentials another way (see [Other Kubernetes Clusters](#other-kubernetes-clusters)). | `true` |
| `serviceAccount.annotations` | Annotations for the service account. The chart renders only the annotation that matches `cloudProvider`. | `eks.amazonaws.com/role-arn: ""` |
| `serviceAccount.name` | Name of the service account | `nullify-k8s-collector-sa` |
| `serviceAccount.namespace` | Namespace of every namespaced object; `""` means the release namespace. Must already exist. | `nullify` |
Expand All @@ -65,7 +66,7 @@ The chart refuses to render when a required value is missing or still a placehol
| `collector.s3.keyPrefix` | S3 key prefix | `k8s-collector` |
| `collector.aws.region` | Region of the **Nullify S3 bucket**, not your cluster. Defaults to the region in `collector.kms.keyArn`; any other value is rejected, because S3 only accepts a KMS key from the bucket's region. | `""` |
| `collector.clusterName` | Cluster name (must match your actual cluster name) | `""` (required) |
| `collector.kms.keyArn` | KMS **key** ARN `arn:aws:kms:<region>:<account-id>:key/<key-id>`. An alias ARN is rejected: IAM ignores aliases for key operations, so uploads would be denied. | `""` (required) |
| `collector.kms.keyArn` | KMS **key** ARN `arn:aws:kms:<region>:<account-id>:key/<key-id>`. An alias ARN is rejected: IAM ignores aliases for key operations, so uploads would be denied. Use the key ARN from the Nullify configure page; if it shows an alias, contact Nullify. | `""` (required) |
| `collector.debug.enabled` | Enable debug logging for troubleshooting | `false` |
| `collector.gke.awsRoleArn` | **GKE only.** Nullify-owned federated AWS IAM role ARN (provided after cluster registration). | `""` |
| `collector.gke.audience` | **GKE only.** Token audience for the projected SA token. Do not change unless Nullify asks you to. | `sts.amazonaws.com` |
Expand Down Expand Up @@ -160,9 +161,11 @@ helm upgrade --install nullify-k8s-collector nullify/nullify-k8s-collector \

### Other Kubernetes Clusters

For clusters outside EKS and GKE, you'll need to provide AWS credentials through
other means:
For clusters outside EKS and GKE, or EKS without IRSA, you'll need to provide AWS
credentials through other means, and set `serviceAccount.requireRoleArn: false`
so the chart renders without the role-arn annotation:

- Using EKS Pod Identity for the collector's ServiceAccount
- Using AWS environment variables in the pod
- Using instance profiles for nodes running on EC2
- Using a solution like Kube2IAM or kiam
Expand All @@ -171,8 +174,10 @@ other means:

- `namespace.create` and `namespace.requireNamespace` are removed, along with the pre-install namespace hook and the `bitnami/kubectl` Job. Existing namespaces are not deleted. New installs need `--create-namespace`, or an existing namespace.
- `collector.dataCollection.*` is removed. The collector never read `EXCLUDE_NAMESPACES`, `INCLUDE_RESOURCES` or `METADATA_ONLY`.
- `collector.clusterName`, `collector.s3.bucket` and `collector.kms.keyArn` now default to `""`, and rendering fails when they are empty or still the old placeholders. The EKS role-arn annotation is validated the same way.
- `collector.clusterName`, `collector.s3.bucket` and `collector.kms.keyArn` now default to `""`, and rendering fails when they are empty or still the old placeholders. The EKS role-arn annotation is validated the same way, unless `serviceAccount.requireRoleArn: false`.
- A KMS alias in `collector.kms.keyArn` now fails to render. Uploads with an alias were already denied `kms:GenerateDataKey`. Use the key ARN from the Nullify configure page; if it shows an alias, contact Nullify.
- `collector.aws.region` now defaults to the KMS key's region instead of `us-east-1`.
- `helm upgrade --reuse-values` from 0.2.0 keeps 0.2.0's defaults, including `collector.aws.region: us-east-1`, so the region check fails unless the key is in `us-east-1`. Upgrade with `--reset-then-reuse-values` (Helm 3.14+) or `-f <your values>`, or set `collector.aws.region` to the key's region.
- `nodeSelector` and `tolerations` now apply to the collector pod; before, only the removed pre-install Job used them.

## Uninstallation
Expand Down Expand Up @@ -204,7 +209,7 @@ If the CronJob is not creating jobs on schedule:

If the job is failing due to S3 access issues:

1. `AccessDenied` on `kms:GenerateDataKey`: `collector.kms.keyArn`, and `NullifyKMSKeyArn` on the CloudFormation stack, must be the key ARN, not an alias.
1. `AccessDenied` on `kms:GenerateDataKey`: `collector.kms.keyArn`, and `NullifyKMSKeyArn` on the CloudFormation stack, must be the key ARN, not an alias. If the Nullify configure page shows an alias, contact Nullify.
2. `AccessDenied` on `sts:AssumeRoleWithWebIdentity`: the service account must be `nullify/nullify-k8s-collector-sa`, and the integration's EKS OIDC provider URL must match the cluster.
3. Wrong-region or redirect errors: `collector.aws.region` must be the Nullify bucket's region (leave it empty).
4. Examine the job logs for detailed error messages.
Expand Down
11 changes: 11 additions & 0 deletions helm-charts/nullify-k8s-collector/ci/eks-govcloud-role.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# EKS in the aws-us-gov partition: the role-arn guard accepts non-commercial partitions.
cloudProvider: aws
collector:
clusterName: example-eks-cluster
s3:
bucket: example-nullify-bucket
kms:
keyArn: arn:aws:kms:us-east-2:123456789012:key/00000000-0000-0000-0000-000000000000
serviceAccount:
annotations:
eks.amazonaws.com/role-arn: arn:aws-us-gov:iam::123456789012:role/AWSIntegration-example-NullifyReadOnlyRole
10 changes: 10 additions & 0 deletions helm-charts/nullify-k8s-collector/ci/eks-no-irsa.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# EKS without IRSA (Pod Identity, instance profile, kube2iam): no role-arn annotation.
cloudProvider: aws
collector:
clusterName: example-eks-cluster
s3:
bucket: example-nullify-bucket
kms:
keyArn: arn:aws:kms:us-east-2:123456789012:key/00000000-0000-0000-0000-000000000000
serviceAccount:
requireRoleArn: false
2 changes: 1 addition & 1 deletion helm-charts/nullify-k8s-collector/templates/NOTES.txt
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ AWS credential is stored in the cluster. No GKE Workload Identity setup required
{{- else }}

## Authentication (EKS IRSA)
- IAM Role: {{ index (.Values.serviceAccount.annotations | default dict) "eks.amazonaws.com/role-arn" }}
- IAM Role: {{ index (.Values.serviceAccount.annotations | default dict) "eks.amazonaws.com/role-arn" | default "none set; the pod needs AWS credentials from outside this chart (serviceAccount.requireRoleArn=false)" }}
- ServiceAccount: system:serviceaccount:{{ include "k8s-collector.namespace" . }}:{{ include "k8s-collector.serviceAccountName" . }}
(the connector CloudFormation trust only allows system:serviceaccount:nullify:nullify-k8s-collector-sa)
{{- end }}
Expand Down
4 changes: 2 additions & 2 deletions helm-charts/nullify-k8s-collector/templates/_helpers.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ is rejected. Call after "k8s-collector.validateCollector".
{{- $keyRegion := index (splitList ":" .Values.collector.kms.keyArn) 3 -}}
{{- $region := .Values.collector.aws.region | default $keyRegion -}}
{{- if ne $region $keyRegion -}}
{{- fail (printf "collector.aws.region %q does not match the region of collector.kms.keyArn (%q). It must be the region of the Nullify S3 bucket, which is the key's region; leave it empty to use the key's region." $region $keyRegion) -}}
{{- fail (printf "collector.aws.region %q does not match the region of collector.kms.keyArn (%q). It must be the region of the Nullify S3 bucket, which is the key's region; leave it empty to use the key's region. An upgrade from chart 0.2.0 with --reuse-values carries that chart's default us-east-1: upgrade with --reset-then-reuse-values, or set collector.aws.region to the key's region." $region $keyRegion) -}}
{{- end -}}
{{- $region -}}
{{- end }}
Expand All @@ -105,7 +105,7 @@ Rejects missing or placeholder values the CronJob cannot run without.
{{- end -}}
{{- $keyArn := required "collector.kms.keyArn is required: the collector refuses to upload without KMS encryption" .Values.collector.kms.keyArn -}}
{{- if not (regexMatch "^arn:aws:kms:[a-z0-9-]+:[0-9]{12}:key/.+" $keyArn) -}}
{{- fail (printf "collector.kms.keyArn must be a KMS key ARN (arn:aws:kms:<region>:<account-id>:key/<key-id>), got %q. A KMS alias does not work: IAM policies ignore alias ARNs for key operations, so the upload is denied kms:GenerateDataKey. Ask Nullify for the key ARN behind the alias." $keyArn) -}}
{{- fail (printf "collector.kms.keyArn must be a KMS key ARN (arn:aws:kms:<region>:<account-id>:key/<key-id>), got %q. A KMS alias does not work: IAM policies ignore alias ARNs for key operations, so the upload is denied kms:GenerateDataKey. Use the KMS key ARN from the Nullify configure page; if it shows an alias, contact Nullify." $keyArn) -}}
{{- end -}}
{{- if eq $provider "gcp" -}}
{{- $_ := required "collector.gke.awsRoleArn is required when cloudProvider is gcp: Nullify provides it after you register the cluster's OIDC issuer" (.Values.collector.gke.awsRoleArn | default .Values.collector.gke.nullifyAwsRoleArn) -}}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@
{{- if .Values.serviceAccount.create -}}
{{- $provider := .Values.cloudProvider | default "aws" -}}
{{- $values := .Values.serviceAccount.annotations | default dict -}}
{{- if eq $provider "aws" -}}
{{- if and (eq $provider "aws") (ne (toString .Values.serviceAccount.requireRoleArn) "false") -}}
{{- $roleArn := index $values "eks.amazonaws.com/role-arn" | default "" | toString -}}
{{- if not (regexMatch "^arn:aws:iam::[0-9]{12}:role/.+" $roleArn) -}}
{{- fail (printf "serviceAccount.annotations.\"eks.amazonaws.com/role-arn\" must be the IAM role ARN from the Nullify configure page (arn:aws:iam::<account-id>:role/AWSIntegration-<CustomerName>-NullifyReadOnlyRole), got %q" $roleArn) -}}
{{- if not (regexMatch "^arn:aws(-us-gov|-cn)?:iam::[0-9]{12}:role/.+" $roleArn) -}}
{{- fail (printf "serviceAccount.annotations.\"eks.amazonaws.com/role-arn\" must be the IAM role ARN from the Nullify configure page (arn:aws:iam::<account-id>:role/AWSIntegration-<CustomerName>-NullifyReadOnlyRole; the aws-us-gov and aws-cn partitions are accepted), got %q. If the pod gets AWS credentials another way (EKS Pod Identity, a node instance profile, kube2iam or kiam), set serviceAccount.requireRoleArn=false." $roleArn) -}}
{{- end -}}
{{- end -}}
{{- $annotations := dict -}}
{{- range $k, $v := $values -}}
{{- if eq $k "eks.amazonaws.com/role-arn" -}}
{{- if eq $provider "aws" -}}
{{- if and (eq $provider "aws") (toString $v) -}}
{{- $_ := set $annotations $k $v -}}
{{- end -}}
{{- else if eq $k "iam.gke.io/gcp-service-account" -}}
Expand Down
5 changes: 5 additions & 0 deletions helm-charts/nullify-k8s-collector/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -136,6 +136,11 @@ serviceAccount:
# e.g. nullify-k8s-collector@your-project.iam.gserviceaccount.com.
# iam.gke.io/gcp-service-account: "YOUR-GCP-SA-EMAIL"
create: true
# cloudProvider=aws: fail the render unless the role-arn annotation above is
# an IAM role ARN (partitions aws, aws-us-gov, aws-cn). Set false only when
# the pod gets AWS credentials another way: EKS Pod Identity, a node
# instance profile, kube2iam or kiam.
requireRoleArn: true
# Keep the default name and namespace on EKS: the connector's CloudFormation
# IRSA trust only allows system:serviceaccount:nullify:nullify-k8s-collector-sa.
name: nullify-k8s-collector-sa
Expand Down
2 changes: 1 addition & 1 deletion tests/helm/nullify-k8s-collector/must-fail/kms-alias.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# expect: collector.kms.keyArn must be a KMS key ARN
# expect: if it shows an alias, contact Nullify
collector:
clusterName: example-eks-cluster
s3:
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# What `helm upgrade --reuse-values` hands 0.4.0 from a 0.2.0 install: 0.2.0's
# defaults (collector.aws.region us-east-1, namespace.create) plus the user's values.
# expect: upgrade with --reset-then-reuse-values
namespace:
create: true
collector:
aws:
region: us-east-1
clusterName: example-eks-cluster
s3:
bucket: example-nullify-bucket
kms:
keyArn: arn:aws:kms:ap-southeast-2:123456789012:key/00000000-0000-0000-0000-000000000000
serviceAccount:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/AWSIntegration-example-NullifyReadOnlyRole