From c3fc931b81af19aab3787f31d388b554d18fd231 Mon Sep 17 00:00:00 2001 From: James Lavoy Date: Mon, 26 Oct 2026 08:46:30 -0600 Subject: [PATCH 1/3] docs(examples): add examples and fix shellscript script keys The shellscript package only runs .sh keys and treats a missing script as success, so the config.yaml keys in examples/simple and examples/interrupt-wait-for-pod never ran. Rename them to config.sh and config_check.sh, add apply scripts to the quickstart example, and point at ghcr.io/nvidia/nodewright-packages, which is where shellscript 1.1.2 is published. Add examples for dependencies, uninstall, ordering, secrets, runtime-required, deployment-policy rollouts, config interrupts, and a real catalog package (copy-fail), each with a README. Rewrite the examples index and link it from docs/README.md and the quickstart. AI assistance: the examples and READMEs were drafted with Claude Code and reviewed by me. Manifests were checked against the operator's Go validation functions; not applied to a cluster. Signed-off-by: James Lavoy --- docs/README.md | 1 + docs/getting-started/quickstart.md | 32 +++-- examples/README.md | 88 +++++++++++-- examples/config-interrupts/README.md | 92 ++++++++++++++ examples/config-interrupts/nodewright.yaml | 70 +++++++++++ examples/copy-fail-mitigation/README.md | 66 ++++++++++ examples/copy-fail-mitigation/nodewright.yaml | 59 +++++++++ examples/dependencies/README.md | 65 ++++++++++ examples/dependencies/nodewright.yaml | 94 ++++++++++++++ examples/interrupt-wait-for-pod/scr.yaml | 8 +- examples/ordering/README.md | 77 ++++++++++++ examples/ordering/nodewrights.yaml | 97 +++++++++++++++ examples/rollout/README.md | 116 ++++++++++++++++++ examples/rollout/deployment-policy.yaml | 66 ++++++++++ examples/rollout/kind-config.yaml | 30 +++++ examples/rollout/nodewright.yaml | 49 ++++++++ examples/runtime-required/README.md | 82 +++++++++++++ examples/runtime-required/nodewright.yaml | 55 +++++++++ examples/runtime-required/workload.yaml | 31 +++++ examples/secrets/README.md | 68 ++++++++++ examples/secrets/nodewright.yaml | 54 ++++++++ examples/simple/scr.yaml | 20 ++- examples/uninstall/README.md | 75 +++++++++++ examples/uninstall/nodewright.yaml | 54 ++++++++ 24 files changed, 1421 insertions(+), 28 deletions(-) create mode 100644 examples/config-interrupts/README.md create mode 100644 examples/config-interrupts/nodewright.yaml create mode 100644 examples/copy-fail-mitigation/README.md create mode 100644 examples/copy-fail-mitigation/nodewright.yaml create mode 100644 examples/dependencies/README.md create mode 100644 examples/dependencies/nodewright.yaml create mode 100644 examples/ordering/README.md create mode 100644 examples/ordering/nodewrights.yaml create mode 100644 examples/rollout/README.md create mode 100644 examples/rollout/deployment-policy.yaml create mode 100644 examples/rollout/kind-config.yaml create mode 100644 examples/rollout/nodewright.yaml create mode 100644 examples/runtime-required/README.md create mode 100644 examples/runtime-required/nodewright.yaml create mode 100644 examples/runtime-required/workload.yaml create mode 100644 examples/secrets/README.md create mode 100644 examples/secrets/nodewright.yaml create mode 100644 examples/uninstall/README.md create mode 100644 examples/uninstall/nodewright.yaml diff --git a/docs/README.md b/docs/README.md index 11856449c..1e43d3535 100644 --- a/docs/README.md +++ b/docs/README.md @@ -8,6 +8,7 @@ This directory contains user and operator documentation for NodeWright. Here you - [Quickstart](getting-started/quickstart.md): Install NodeWright and run a hello-world package on one node. - [Installation](getting-started/installation.md): Install NodeWright via Helm. - [Migration from Skyhook](getting-started/migration.md): Transition guide from Skyhook to NodeWright. +- [Examples](../examples/README.md): Runnable manifests for dependencies, uninstall, ordering, secrets, runtime-required, deployment policies, and interrupts. ## Architecture diff --git a/docs/getting-started/quickstart.md b/docs/getting-started/quickstart.md index 9d72a11e6..006ef5d3a 100644 --- a/docs/getting-started/quickstart.md +++ b/docs/getting-started/quickstart.md @@ -79,14 +79,26 @@ spec: nodewright.nvidia.com/quickstart: "true" packages: hello-world: - version: 1.1.0 - image: ghcr.io/nvidia/skyhook-packages/shellscript + version: 1.1.2 + image: ghcr.io/nvidia/nodewright-packages/shellscript configMap: - config.yaml: |- + # Stages run in order: apply first, then config. Each stage runs its + # script, then its _check script, and the stage is only done once the + # check passes. + apply.sh: |- + #!/bin/bash + sleep 1 + echo "Hello, apply!" + apply_check.sh: |- + #!/bin/bash + sleep 1 + echo "Hello, apply check!" + # Runs only after apply and apply_check have succeeded. + config.sh: |- #!/bin/bash sleep 1 echo "Hello, config!" - config_check.yaml: |- + config_check.sh: |- #!/bin/bash sleep 1 echo "Hello, config check!" @@ -135,7 +147,7 @@ Four parts of that manifest did all the work. Every field is covered in | `nodeSelectors` | Chose which nodes to act on. Empty means *all* nodes — always set it deliberately. | | `packages` | The unit of work. The map key (`hello-world`) is the package name. | | `image` + `version` | `version` is the tag **and** the operator's ordering key, so it must be semver and the tag must not be inlined into `image`. | -| `configMap` | Configuration handed to the package. The `shellscript` package treats these keys as scripts, which is why no image build was needed. | +| `configMap` | Configuration handed to the package. The `shellscript` package runs these keys as scripts, which is why no image build was needed. Its key names are fixed (`apply.sh`, `config.sh`, and so on); a key it does not recognise is skipped without an error. | Two knobs you did not set are worth knowing before you go near a real cluster: `interruptionBudget` defaults to **100%** — every matching node at once — and @@ -148,13 +160,15 @@ it declared no `interrupt`, it stopped there and the node was never cordoned or drained. [Lifecycle of a NodeWright](../architecture/lifecycle.md) has the whole picture. -The part worth internalising is what your two configMap keys were: +The part worth internalising is what the `config` pair of configMap keys were: ```text -config.yaml → the config stage's WORK step -config_check.yaml → the config stage's CHECK step +config.sh → the config stage's WORK step +config_check.sh → the config stage's CHECK step ``` +`apply.sh` and `apply_check.sh` were the same pairing for the apply stage. + **Every work step has a paired check step, and the stage is not done until the check passes.** That is not a naming convention — the operator runs them as consecutive init containers in the same pod, so Kubernetes itself stops at the @@ -185,4 +199,4 @@ helm uninstall nodewright --namespace nodewright - [Lifecycle of a NodeWright](../architecture/lifecycle.md) — how a rollout actually runs - [Interrupt Flow](../architecture/interrupt-flow.md) — cordon, drain, reboot - [Deployment Policy](../user-guide/deployment-policy.md) — shaping a rollout across many nodes -- [`examples/`](../../examples/) — including an interrupt example with real workloads +- [`examples/`](../../examples/) — runnable examples for dependencies, uninstall, ordering, secrets, runtime-required, rollouts, and interrupts diff --git a/examples/README.md b/examples/README.md index 8223378ed..9973e8b2d 100644 --- a/examples/README.md +++ b/examples/README.md @@ -1,16 +1,82 @@ # NodeWright Examples -This directory contains sample manifests and usage patterns for NodeWright. Use these examples to learn how to configure, test, and extend NodeWright in your own clusters. +Manifests you can apply to a cluster to see NodeWright do something specific. Each +directory has its own README with the commands to run and what you should see. -## Included Examples +If you have not used NodeWright before, start with [`simple/`](simple/), then work +down the table. -- `debug_pod.yaml`: Example manifest for launching a debug pod in your cluster. - - Replace the `{{ set hostname }}` with the hostname you want to put the debug pod onto - - `kubectl apply -f examples/debug_pod.yaml` - - Once launched do `kubectl exec -ti debug-pod sh -c 'chroot /host bash'` - - You will then be on the host as root - - Once done, exit and run `kubectl delete po/debug-pod` to remove the pod -- `simple/`: The hello-world NodeWright used by the [quickstart](../docs/getting-started/quickstart.md). -- `interrupt-wait-for-pod/`: Example showing how to use interrupts and wait-for-pod logic with NodeWright. +| Example | What it shows | Where to run it | +|---|---|---| +| [`simple/`](simple/) | The hello world: one package, no interrupt. Used by the [quickstart](../docs/getting-started/quickstart.md). | Any cluster | +| [`dependencies/`](dependencies/) | `dependsOn`: a diamond of packages, with the run order read back from the host | Any cluster | +| [`uninstall/`](uninstall/) | Reversing a package: `uninstall.enabled` and `uninstall.apply`, cancelling, and delete-time cleanup | Any cluster | +| [`ordering/`](ordering/) | `priority`, `sequencing: all` as a cluster-wide barrier, and `pause` as a brake | Any cluster | +| [`secrets/`](secrets/) | Passing a Kubernetes Secret to a package through `env`, without logging it | Any cluster | +| [`runtime-required/`](runtime-required/) | Gating workloads behind a node being ready, with the runtime-required taint | Any cluster | +| [`rollout/`](rollout/) | A `DeploymentPolicy`: canary, production, and default compartments, and a stopped compartment | Six or more workers; a kind config is included | +| [`interrupt-wait-for-pod/`](interrupt-wait-for-pod/) | `podNonInterruptLabels`: a reboot interrupt that waits for your workload to finish | Three nodes, EKS node group labels | +| [`config-interrupts/`](config-interrupts/) | Matching the interrupt to the change with `configInterrupts`, using the real `tuning` package | Disposable real nodes | +| [`copy-fail-mitigation/`](copy-fail-mitigation/) | A real package from the catalog, rolled out the production way | Real nodes | +| [`debug_pod.yaml`](debug_pod.yaml) | A privileged pod with the node's filesystem mounted, for looking at what a package did | Any cluster | -Feel free to use, modify, or extend these examples for your own use cases! +## Conventions + +- **Nothing happens until you opt a node in.** Every example selects nodes by the + label `nodewright.nvidia.com/example=`, which no node carries by default. + A NodeWright with an empty `nodeSelectors` matches *every* node in the cluster, so + these examples never rely on that. Label one node and the blast radius is one node. +- **Packages come from [`nodewright-packages`](https://github.com/NVIDIA/nodewright-packages).** + Most examples use the `shellscript` package, which runs the scripts you put in + `configMap`. Its script names are fixed: `apply.sh`, `apply_check.sh`, `config.sh`, + `config_check.sh`, `post_interrupt.sh`, `post_interrupt_check.sh`, `uninstall.sh`, + and `uninstall_check.sh`. A script that is missing is skipped with a message and + counts as success, so a misspelled key does nothing rather than failing. +- **Every step has a check.** A stage is done when its `*_check` script passes, not + when its work script exits. +- **Scripts run on the host**, as root, in a chroot of the node's filesystem. The + examples write only under `/etc/nodewright-example/` so cleanup is one `rm -rf`. +- **Clean up after yourself.** Deleting a NodeWright does not undo what its packages + did to a host. Each README ends with a clean-up section, and says what stays behind. + +The commands assume the operator is installed in the `nodewright` namespace, the +default for new installs. Installs that predate the rename are in `skyhook`. + +## Reading files on a node + +Several examples write a file to the host. To read it, run a debug pod pinned to the +node: + +```bash +NODE= +sed "s/{{ some hostname }}/$NODE/" examples/debug_pod.yaml | kubectl apply -f - +kubectl exec -ti debug-pod -- chroot /host cat /etc/nodewright-example/ +kubectl exec -ti debug-pod -- chroot /host bash # or an interactive root shell +kubectl delete pod debug-pod +``` + +The pod is privileged and mounts the whole host filesystem. Delete it when you are done. + +## Reading what NodeWright did + +```bash +kubectl get nodewright -w # overall status +kubectl nodewright node status # every NodeWright on a node +kubectl nodewright package status --nodewright -o wide +kubectl nodewright package logs --nodewright --stage apply +kubectl get nodewright -o jsonpath='{.status.conditions}' # why nothing is happening +``` + +The authoritative record of each package is an annotation on the node, +`nodewright.nvidia.com/nodeState_`, not the NodeWright's status. Status, +state, and stage are three different things; see +[Operator Status](../docs/architecture/operator-status.md). + +## Adding an example + +Keep it to one idea, safe to apply to a single node, and honest about what it leaves +on the host. Use the opt-in label convention above, give every YAML file the +license header (`make fmt` adds it), and link the docs page that explains the +feature. Check it with `kubectl apply --dry-run=server -f` against a cluster that has the +operator installed, so the admission webhook validates it: an example the webhook +rejects is worse than none. diff --git a/examples/config-interrupts/README.md b/examples/config-interrupts/README.md new file mode 100644 index 000000000..a21e249d6 --- /dev/null +++ b/examples/config-interrupts/README.md @@ -0,0 +1,92 @@ +# Example: matching the interrupt to the change + +An interrupt is the disruptive step after a package applies: a service restart or +a reboot. It means the node is cordoned and drained first, so you want the +smallest one that does the job. + +This example uses the real [`tuning`](https://github.com/NVIDIA/nodewright-packages/tree/main/tuning) +package to change kernel and ulimit settings, and gives each setting the interrupt +it actually needs: + +| configMap key | Changes | Interrupt | +|---|---|---| +| `sysctl.conf` | kernel parameters | `restartAllServices` | +| `ulimit.conf` | per-process limits | none: no `configInterrupts` entry | +| `grub.conf` (commented out) | kernel command line | `reboot` | + +`interrupt` is what runs when the package is first applied. `configInterrupts` +takes over for later edits, and runs only the interrupt for the key that changed. + +> **This changes real host settings and restarts services.** Use disposable nodes. + +## Run it + +```bash +NODE= +kubectl label node "$NODE" nodewright.nvidia.com/example=config-interrupts +kubectl apply -f examples/config-interrupts/nodewright.yaml +kubectl get nodewright config-interrupts-demo -w +``` + +Because the package declares an interrupt, the node is cordoned and drained +before it runs. In another terminal: + +```bash +kubectl get node "$NODE" -w +kubectl nodewright package status tuning --nodewright config-interrupts-demo -o wide +``` + +You will see the package move through `apply`, `config`, `interrupt`, and +`post-interrupt`, and the node uncordon when it finishes. + +## Change one setting at a time + +**Edit `ulimit.conf`.** Change `nofile=1048576` to another value and re-apply. The +package re-runs `config` and finishes. The node is never cordoned or drained, +because no `configInterrupts` entry matches `ulimit.conf`. + +```bash +kubectl patch nodewright config-interrupts-demo --type=merge \ + -p '{"spec":{"packages":{"tuning":{"configMap":{"ulimit.conf":"nofile=524288"}}}}}' +``` + +**Edit `sysctl.conf`.** Change `vm.max_map_count`. This time the node is cordoned, +drained, and services are restarted, but it is not rebooted. + +**Add `grub.conf`.** Uncomment `grub.conf` in both places and re-apply. Now the node +reboots. + +> A `reboot` anywhere wins. If several packages become due on one node in the same +> pass, their interrupts merge into one, ranked `reboot` over `restartAllServices` +> over `service` over `noop`. The node reboots once instead of restarting services +> first. + +## Related knobs, set in the manifest + +- `interruptionBudget.count: 1` means only one node is ever cordoned and drained at + a time. +- `drainConfig.timeout` bounds a drain that would otherwise wait forever on a pod + that will not leave. When it expires the node goes `erroring`. +- To keep specific pods from being evicted, see + [`interrupt-wait-for-pod`](../interrupt-wait-for-pod/demo.md). + +## Clean up + +`tuning` writes drop-in files, so it can take them out again. Uninstall first, then +delete, otherwise the settings stay on the host: + +```bash +kubectl patch nodewright config-interrupts-demo --type=merge \ + -p '{"spec":{"packages":{"tuning":{"uninstall":{"apply":true}}}}}' +kubectl get nodewright config-interrupts-demo -w # wait for complete +kubectl delete -f examples/config-interrupts/nodewright.yaml +kubectl label node "$NODE" nodewright.nvidia.com/example- +``` + +See the [`uninstall`](../uninstall/README.md) example for how that workflow works. + +## Related + +- [Interrupt Flow](../../docs/architecture/interrupt-flow.md): cordon, drain, + interrupt, uncordon, in full +- [The NodeWright Custom Resource: `configInterrupts`](../../docs/user-guide/custom-resource.md#configmap-and-configinterrupts) diff --git a/examples/config-interrupts/nodewright.yaml b/examples/config-interrupts/nodewright.yaml new file mode 100644 index 000000000..6cd04ca4e --- /dev/null +++ b/examples/config-interrupts/nodewright.yaml @@ -0,0 +1,70 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Matches the interrupt to the change, using the `tuning` package. +# +# - `interrupt` is what runs when the package is first applied. +# - `configInterrupts` is what runs when one specific configMap key changes +# afterwards. A key with no entry (ulimit.conf here) changes with no interrupt +# at all, so the node is never cordoned or drained for it. +# +# This changes real host settings and restarts services. Use disposable nodes. +# +# Opt a node in with: +# kubectl label node nodewright.nvidia.com/example=config-interrupts + +apiVersion: nodewright.nvidia.com/v1alpha1 +kind: NodeWright +metadata: + name: config-interrupts-demo +spec: + nodeSelectors: + matchLabels: + nodewright.nvidia.com/example: config-interrupts + + # One node at a time: an interrupt cordons and drains its node. + interruptionBudget: + count: 1 + + # Give up on a drain that is stuck instead of waiting forever. On timeout the + # node goes `erroring` and the package does not run on it. + drainConfig: + timeout: 10m + + packages: + tuning: + version: 1.1.7 + image: ghcr.io/nvidia/nodewright-packages/tuning + interrupt: + type: restartAllServices + # tuning writes drop-in files, so it can remove them again; see README.md. + uninstall: + enabled: true + apply: false + configInterrupts: + sysctl.conf: + type: restartAllServices + # Uncomment grub.conf below, and this entry, to see a reboot interrupt. + # grub.conf: + # type: reboot + configMap: + sysctl.conf: |- + vm.max_map_count=262144 + fs.inotify.max_user_watches=524288 + ulimit.conf: |- + nofile=1048576 + # grub.conf: |- + # hugepagesz=2M + # hugepages=1024 diff --git a/examples/copy-fail-mitigation/README.md b/examples/copy-fail-mitigation/README.md new file mode 100644 index 000000000..0df787425 --- /dev/null +++ b/examples/copy-fail-mitigation/README.md @@ -0,0 +1,66 @@ +# Example: rolling out a real package + +The other examples use throwaway scripts. This one applies a package from the +[NodeWright package catalog](https://github.com/NVIDIA/nodewright-packages), +`copy-fail`, the way you would roll it out for real. + +`copy-fail` is the temporary host mitigation for CVE-2026-31431. It blacklists the +`algif_aead` kernel module through `/etc/modprobe.d/disable-algif.conf` and makes a +best-effort `rmmod`. It does not reboot the node. See the +[package README](https://github.com/NVIDIA/nodewright-packages/tree/main/copy-fail) +for what it does and does not cover. + +What makes the manifest production-shaped rather than a demo: + +- **Selection is deliberate.** It names its nodes with a label and excludes the + control plane with a `matchExpressions` rule. An empty `nodeSelectors` would match + every node. +- **The rollout is bounded.** `interruptionBudget.percent: 25` caps how many nodes + run at once. With no interrupt nothing is drained, but an unset budget still means + 100%. +- **There is a way back.** `uninstall.enabled: true` lets you remove the + mitigation by flipping `apply` once your distribution ships a fixed kernel. + +## Run it + +Start on one node: + +```bash +NODE= +kubectl label node "$NODE" nodewright.nvidia.com/example=copy-fail +kubectl apply -f examples/copy-fail-mitigation/nodewright.yaml +kubectl get nodewright copy-fail-mitigation -w +``` + +Check the result on the node. From a [debug pod](../README.md#reading-files-on-a-node): + +```bash +chroot /host cat /etc/modprobe.d/disable-algif.conf +chroot /host sh -c 'lsmod | grep algif_aead || echo "algif_aead not loaded"' +``` + +If `config-check` fails because the module is still in use, the node reads +`erroring` until it reboots. That is the package being honest that the node is +still exposed. Reboot on your own schedule, or set `ALLOW_LOADED_MODULE` (commented +in the manifest) to accept the risk until then. + +Widen the rollout by labelling more nodes, or replace the label selector with one +that matches the fleet. + +## Retire it + +When fixed kernels are deployed: + +```bash +kubectl patch nodewright copy-fail-mitigation --type=merge \ + -p '{"spec":{"packages":{"copy-fail":{"uninstall":{"apply":true}}}}}' +``` + +Wait for `complete`, then delete the NodeWright. See the +[`uninstall`](../uninstall/README.md) example. + +## On read-only host filesystems + +On nodes with a read-only root such as GKE Container-Optimized OS, the operator +needs a writable directory for the agent. The package README covers the Helm +values (`copyDirRoot`, `reapplyOnReboot`). diff --git a/examples/copy-fail-mitigation/nodewright.yaml b/examples/copy-fail-mitigation/nodewright.yaml new file mode 100644 index 000000000..44faad11a --- /dev/null +++ b/examples/copy-fail-mitigation/nodewright.yaml @@ -0,0 +1,59 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# A real package from the NodeWright package catalog, applied the way you would +# apply it to production: a bounded rollout that leaves the control plane alone. +# +# The `copy-fail` package disables the algif_aead kernel module as the temporary +# mitigation for CVE-2026-31431. It writes /etc/modprobe.d/disable-algif.conf and +# makes a best-effort rmmod. It does not reboot the node. +# +# Opt nodes in with: +# kubectl label node nodewright.nvidia.com/example=copy-fail + +apiVersion: nodewright.nvidia.com/v1alpha1 +kind: NodeWright +metadata: + name: copy-fail-mitigation +spec: + nodeSelectors: + matchLabels: + nodewright.nvidia.com/example: copy-fail + matchExpressions: + - key: node-role.kubernetes.io/control-plane + operator: DoesNotExist + + # No interrupt is declared, so nothing is drained and an unset budget would + # mean every matching node at once. Bound it anyway. + interruptionBudget: + percent: 25 + + packages: + copy-fail: + version: 1.0.0 + image: ghcr.io/nvidia/nodewright-packages/copy-fail + + # config-check fails while algif_aead is still loaded, so a node that could + # not unload the module reads `erroring` until it reboots. Uncomment to + # accept that until your next planned reboot. + # env: + # - name: ALLOW_LOADED_MODULE + # value: "true" + + # Lets you remove the mitigation later by flipping `apply` to true, + # once your distribution ships a fixed kernel. + uninstall: + enabled: true + apply: false diff --git a/examples/dependencies/README.md b/examples/dependencies/README.md new file mode 100644 index 000000000..f04d8a23c --- /dev/null +++ b/examples/dependencies/README.md @@ -0,0 +1,65 @@ +# Example: package dependencies + +Packages in one NodeWright form a graph. `dependsOn` says which packages must be +complete before another one starts, and everything with no unmet dependency runs +in parallel. + +This example builds a diamond: `base`, then `service-a` and `service-b` together, +then `report`. Each package appends to a log on the host, so afterwards you can +read off exactly what ran when. + +## Run it + +```bash +NODE= +kubectl label node "$NODE" nodewright.nvidia.com/example=dependencies +kubectl apply -f examples/dependencies/nodewright.yaml +kubectl get nodewright dependencies-demo -w +``` + +Wait for `complete`, then read the `report` package's output, which is the log: + +```bash +kubectl nodewright package logs report --nodewright dependencies-demo --stage apply +``` + +```text +--- order in which the packages ran on --- +12:00:01 base: start +12:00:06 base: done +12:00:09 service-a: start +12:00:09 service-b: start +12:00:19 service-a: done +12:00:19 service-b: done +``` + +Your timestamps will differ. What matters: + +- Nothing starts before `base: done`. +- `service-a` and `service-b` start together, because neither depends on the other. +- `report` runs only after both are done. + +## Things to try + +- **Break the graph.** Change `report` to depend on a package that is not defined, + or make `base` depend on `report`. The webhook rejects both when you apply: the + graph has to be a valid DAG. +- **Serialize it.** Add `serial: true` under `spec`. Packages that were running in + parallel now run one at a time per node. That is for packages that fight over + the same host resource; it does not roll out one node at a time. + +## Clean up + +```bash +kubectl delete -f examples/dependencies/nodewright.yaml +kubectl label node "$NODE" nodewright.nvidia.com/example- +``` + +Deleting the NodeWright does not undo what its packages did. The log stays on the +host; remove it from a [debug pod](../README.md#reading-files-on-a-node) with +`chroot /host rm -rf /etc/nodewright-example/dependencies`. + +## Related + +- [The NodeWright Custom Resource: `dependsOn`](../../docs/user-guide/custom-resource.md#dependson) +- [Lifecycle of a NodeWright](../../docs/architecture/lifecycle.md) diff --git a/examples/dependencies/nodewright.yaml b/examples/dependencies/nodewright.yaml new file mode 100644 index 000000000..9ca4fcda7 --- /dev/null +++ b/examples/dependencies/nodewright.yaml @@ -0,0 +1,94 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# A diamond-shaped package DAG: +# +# base +# / \ +# service-a service-b <- run in parallel, once base is complete +# \ / +# report <- runs last, prints the shared log +# +# Opt a node in with: +# kubectl label node nodewright.nvidia.com/example=dependencies + +apiVersion: nodewright.nvidia.com/v1alpha1 +kind: NodeWright +metadata: + name: dependencies-demo +spec: + nodeSelectors: + matchLabels: + nodewright.nvidia.com/example: dependencies + packages: + base: + version: 1.0.0 + image: ghcr.io/nvidia/nodewright-packages/shellscript + configMap: + apply.sh: |- + #!/bin/bash + mkdir -p /etc/nodewright-example/dependencies + echo "$(date -u +%T) base: start" >> /etc/nodewright-example/dependencies/order.log + sleep 5 + echo "$(date -u +%T) base: done" >> /etc/nodewright-example/dependencies/order.log + apply_check.sh: |- + #!/bin/bash + grep -q "base: done" /etc/nodewright-example/dependencies/order.log + + service-a: + version: 1.0.0 + image: ghcr.io/nvidia/nodewright-packages/shellscript + dependsOn: + base: 1.0.0 + configMap: + apply.sh: |- + #!/bin/bash + echo "$(date -u +%T) service-a: start" >> /etc/nodewright-example/dependencies/order.log + sleep 10 + echo "$(date -u +%T) service-a: done" >> /etc/nodewright-example/dependencies/order.log + apply_check.sh: |- + #!/bin/bash + grep -q "service-a: done" /etc/nodewright-example/dependencies/order.log + + service-b: + version: 1.0.0 + image: ghcr.io/nvidia/nodewright-packages/shellscript + dependsOn: + base: 1.0.0 + configMap: + apply.sh: |- + #!/bin/bash + echo "$(date -u +%T) service-b: start" >> /etc/nodewright-example/dependencies/order.log + sleep 10 + echo "$(date -u +%T) service-b: done" >> /etc/nodewright-example/dependencies/order.log + apply_check.sh: |- + #!/bin/bash + grep -q "service-b: done" /etc/nodewright-example/dependencies/order.log + + report: + version: 1.0.0 + image: ghcr.io/nvidia/nodewright-packages/shellscript + dependsOn: + service-a: 1.0.0 + service-b: 1.0.0 + configMap: + apply.sh: |- + #!/bin/bash + echo "--- order in which the packages ran on $(hostname) ---" + cat /etc/nodewright-example/dependencies/order.log + apply_check.sh: |- + #!/bin/bash + grep -q "service-a: done" /etc/nodewright-example/dependencies/order.log && + grep -q "service-b: done" /etc/nodewright-example/dependencies/order.log diff --git a/examples/interrupt-wait-for-pod/scr.yaml b/examples/interrupt-wait-for-pod/scr.yaml index 779378a53..f62f93602 100644 --- a/examples/interrupt-wait-for-pod/scr.yaml +++ b/examples/interrupt-wait-for-pod/scr.yaml @@ -35,17 +35,17 @@ spec: app: nodewright-demo-workload packages: baz: - version: 1.1.0 - image: ghcr.io/nvidia/skyhook-packages/shellscript + version: 1.1.2 + image: ghcr.io/nvidia/nodewright-packages/shellscript interrupt: type: reboot # type: service # services: # - cron configMap: - config.yaml: |- + config.sh: |- #!/bin/bash sleep 100 - config_check.yaml: |- + config_check.sh: |- #!/bin/bash sleep 100 diff --git a/examples/ordering/README.md b/examples/ordering/README.md new file mode 100644 index 000000000..34aa99695 --- /dev/null +++ b/examples/ordering/README.md @@ -0,0 +1,77 @@ +# Example: running NodeWrights in a fixed order + +When several NodeWrights target the same nodes, `priority` decides the order and +`sequencing` decides how strictly it is held. This example has three: + +| NodeWright | `priority` | `sequencing` | Meaning | +|---|---|---|---| +| `ordering-infra` | 10 | `node` (default) | Each node moves on as soon as it finishes here | +| `ordering-config` | 100 | `all` | A cluster-wide barrier: **no** node starts priority 200 until **every** node has finished this one | +| `ordering-workload` | 200 | `node` | Runs last | + +Lower numbers run first. NodeWrights with the same priority run in name order. +The numbers follow the [recommended buckets](../../docs/architecture/ordering.md#recommended-priority-buckets): +1-99 infrastructure, 100-199 configuration, 200 and up user-level work. + +## Run it + +```bash +NODE= +kubectl label node "$NODE" nodewright.nvidia.com/example=ordering +kubectl apply -f examples/ordering/nodewrights.yaml +kubectl get nodewright +``` + +`ordering-config` carries a `nodewright.nvidia.com/pause: "true"` annotation, so the +run stops there. `ordering-infra` completes, `ordering-config` reads `paused`, and +`ordering-workload` waits behind it. **Pause stops the paused NodeWright and +everything after it** on that node, which is what makes it a safe brake. + +Release it: + +```bash +kubectl annotate nodewright ordering-config nodewright.nvidia.com/pause=false --overwrite +kubectl get nodewright -w +``` + +Now `ordering-config` runs, then `ordering-workload`. The last package prints the +log of what ran, in order: + +```bash +kubectl nodewright package logs workload --nodewright ordering-workload --stage apply +``` + +```text +--- order in which the NodeWrights ran on --- +12:00:01 10 ordering-infra +12:00:30 100 ordering-config +12:00:41 200 ordering-workload +``` + +Pausing is a flag on the resource, so toggling it does not change the spec's +generation. `disable` is the sibling: it skips a NodeWright and lets the ones +after it carry on. + +## Use `sequencing: all` sparingly + +A barrier is only as fast as its slowest node. If one node never finishes the +`all` NodeWright, every node waits at it indefinitely. That is the right trade when +the next stage truly needs the whole fleet in the same state, and a trap +otherwise. Never combine it with `runtimeRequired: true`: a single unhealthy node +would keep every node tainted. + +## Clean up + +```bash +kubectl delete -f examples/ordering/nodewrights.yaml +kubectl label node "$NODE" nodewright.nvidia.com/example- +``` + +The log stays on the host. Remove it from a +[debug pod](../README.md#reading-files-on-a-node) with +`chroot /host rm -rf /etc/nodewright-example/ordering`. + +## Related + +- [Strict Ordering](../../docs/architecture/ordering.md): priority, sequencing, + pause and disable, and the deadlock cases diff --git a/examples/ordering/nodewrights.yaml b/examples/ordering/nodewrights.yaml new file mode 100644 index 000000000..fd65a5f23 --- /dev/null +++ b/examples/ordering/nodewrights.yaml @@ -0,0 +1,97 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Three NodeWrights that must run in a fixed order, using the priority buckets +# from docs/architecture/ordering.md. +# +# ordering-infra priority 10 per-node: each node moves on as soon as it is done +# ordering-config priority 100 sequencing: all, so no node starts priority 200 +# until every node has finished this one +# ordering-workload priority 200 runs last +# +# ordering-config starts paused so you can watch the order hold. README.md +# unpauses it. +# +# Opt nodes in with: +# kubectl label node nodewright.nvidia.com/example=ordering + +apiVersion: nodewright.nvidia.com/v1alpha1 +kind: NodeWright +metadata: + name: ordering-infra +spec: + priority: 10 + nodeSelectors: + matchLabels: + nodewright.nvidia.com/example: ordering + packages: + infra: + version: 1.0.0 + image: ghcr.io/nvidia/nodewright-packages/shellscript + configMap: + apply.sh: |- + #!/bin/bash + mkdir -p /etc/nodewright-example/ordering + echo "$(date -u +%T) 10 ordering-infra" >> /etc/nodewright-example/ordering/order.log + apply_check.sh: |- + #!/bin/bash + grep -q "ordering-infra" /etc/nodewright-example/ordering/order.log +--- +apiVersion: nodewright.nvidia.com/v1alpha1 +kind: NodeWright +metadata: + name: ordering-config + annotations: + nodewright.nvidia.com/pause: "true" +spec: + priority: 100 + sequencing: all + nodeSelectors: + matchLabels: + nodewright.nvidia.com/example: ordering + packages: + config: + version: 1.0.0 + image: ghcr.io/nvidia/nodewright-packages/shellscript + configMap: + apply.sh: |- + #!/bin/bash + echo "$(date -u +%T) 100 ordering-config" >> /etc/nodewright-example/ordering/order.log + apply_check.sh: |- + #!/bin/bash + grep -q "ordering-config" /etc/nodewright-example/ordering/order.log +--- +apiVersion: nodewright.nvidia.com/v1alpha1 +kind: NodeWright +metadata: + name: ordering-workload +spec: + priority: 200 + nodeSelectors: + matchLabels: + nodewright.nvidia.com/example: ordering + packages: + workload: + version: 1.0.0 + image: ghcr.io/nvidia/nodewright-packages/shellscript + configMap: + apply.sh: |- + #!/bin/bash + echo "$(date -u +%T) 200 ordering-workload" >> /etc/nodewright-example/ordering/order.log + echo "--- order in which the NodeWrights ran on $(hostname) ---" + cat /etc/nodewright-example/ordering/order.log + apply_check.sh: |- + #!/bin/bash + grep -q "ordering-workload" /etc/nodewright-example/ordering/order.log diff --git a/examples/rollout/README.md b/examples/rollout/README.md new file mode 100644 index 000000000..c07cd24c7 --- /dev/null +++ b/examples/rollout/README.md @@ -0,0 +1,116 @@ +# Example: a staged rollout with a DeploymentPolicy + +An `interruptionBudget` caps how many nodes run at once. A `DeploymentPolicy` +goes further: it splits the fleet into **compartments**, each with its own batch +strategy and its own tolerance for failure, so one rollout can be cautious with +some nodes and quick with others. + +This example sorts six workers into three groups: + +| Group | Nodes | Rolls out | Stops when | +|---|---|---|---| +| `canary` | 1 | one at a time | the first batch fails | +| `production` | 3 | 1 node, then 2, growing by one per good batch | two batches in a row fail | +| default | 2 | two at a time | the first batch fails | + +Nodes that match no compartment fall into the policy's `default`. + +## Prerequisites + +Six or more worker nodes. To make a throwaway cluster: + +```bash +kind create cluster --name nodewright-examples --config examples/rollout/kind-config.yaml +``` + +and [install NodeWright](../../docs/getting-started/quickstart.md#1-install-the-operator). + +## Run it + +Opt every worker in, then sort them into compartments: + +```bash +WORKERS=($(kubectl get nodes -l '!node-role.kubernetes.io/control-plane' -o name | cut -d/ -f2)) + +for n in "${WORKERS[@]}"; do + kubectl label node "$n" nodewright.nvidia.com/example=rollout +done +kubectl label node "${WORKERS[0]}" nodewright.nvidia.com/example-tier=canary +for n in "${WORKERS[@]:1:3}"; do + kubectl label node "$n" nodewright.nvidia.com/example-tier=production +done +# the last two carry no tier label and fall into `default` +``` + +Create the policy before the NodeWright, because the webhook rejects a reference +to a policy that does not exist: + +```bash +kubectl apply -f examples/rollout/deployment-policy.yaml +kubectl apply -f examples/rollout/nodewright.yaml +``` + +Watch the nodes move through in batches (each takes about 20 seconds): + +```bash +kubectl get nodewright rollout-demo -o jsonpath='{.status.nodeStatus}' ; echo +watch -n 5 "kubectl get nodewright rollout-demo -o jsonpath='{.status.compartmentStatuses}'" +``` + +`status.compartmentStatuses` has one entry per compartment, including its batch +state. Compartments run independently, so canary and production move at the same +time. + +## See a compartment stop + +The package fails on any node that has `/etc/nodewright-example/rollout/fail`. +Make the canary node fail, then run the rollout again. From a +[debug pod on the canary node](../README.md#reading-files-on-a-node), run +`chroot /host sh -c 'mkdir -p /etc/nodewright-example/rollout && touch /etc/nodewright-example/rollout/fail'`. +A completed package is not re-run on its own, so reset the NodeWright to send it +through again: + +```bash +kubectl nodewright reset rollout-demo --confirm +``` + +The canary node retries, then reads `erroring`. Its batch counts as failed, and +with `failureThreshold: 1` the **canary compartment stops**. The `production` and +default compartments keep going: stopping applies to the compartment that failed, +not the whole NodeWright. + +A stopped compartment stays stopped until its batch state is reset. After +removing the marker file, `reset` clears the failed node and the batch state +together, and the rollout starts over: + +```bash +kubectl nodewright reset rollout-demo --confirm +``` + +## Two things that surprise people + +- **Once a compartment has a `strategy`, its `budget` no longer caps how many + nodes run at once.** The strategy sets the batch size. To cap concurrency, use + `fixed` with `initialBatch: N`. The budget still breaks ties for a node that + matches several compartments. +- **`safetyLimit` defaults to 50.** Past half of a compartment, the rollout commits + to finishing and stops enforcing `failureThreshold`. These manifests set it to + 100 so a failure stops a rollout at any point. + +## Clean up + +```bash +kubectl delete -f examples/rollout/nodewright.yaml +kubectl delete -f examples/rollout/deployment-policy.yaml +for n in "${WORKERS[@]}"; do + kubectl label node "$n" nodewright.nvidia.com/example- nodewright.nvidia.com/example-tier- +done +kind delete cluster --name nodewright-examples # if you made one +``` + +## Related + +- [Deployment Policy and Compartments](../../docs/user-guide/deployment-policy.md): + strategies, thresholds, overlapping selectors, batch state +- [Strict Ordering](../../docs/architecture/ordering.md#node-order-within-a-rollout): + `SKYHOOK_NODE_ORDER`, the position of each node in a rollout diff --git a/examples/rollout/deployment-policy.yaml b/examples/rollout/deployment-policy.yaml new file mode 100644 index 000000000..4e2bc36d6 --- /dev/null +++ b/examples/rollout/deployment-policy.yaml @@ -0,0 +1,66 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# A rollout shaped per group of nodes. Each compartment is a named node selector +# with its own batch strategy; they roll out independently of one another. +# +# canary one node at a time, stop on the first failed batch +# production starts at one node and grows by one after each good batch +# default every node that matches no compartment: two at a time +# +# `safetyLimit: 100` means failureThreshold is enforced for the whole rollout. +# At the default of 50 the rollout commits to finishing once half the +# compartment has been processed, whatever failureThreshold says. + +apiVersion: nodewright.nvidia.com/v1alpha1 +kind: DeploymentPolicy +metadata: + name: staged-rollout +spec: + default: + budget: + percent: 100 + strategy: + fixed: + initialBatch: 2 + batchThreshold: 100 + failureThreshold: 1 + safetyLimit: 100 + compartments: + - name: canary + selector: + matchLabels: + nodewright.nvidia.com/example-tier: canary + budget: + count: 1 + strategy: + fixed: + initialBatch: 1 + batchThreshold: 100 + failureThreshold: 1 + safetyLimit: 100 + - name: production + selector: + matchLabels: + nodewright.nvidia.com/example-tier: production + budget: + percent: 50 + strategy: + linear: + initialBatch: 1 + delta: 1 + batchThreshold: 100 + failureThreshold: 2 + safetyLimit: 100 diff --git a/examples/rollout/kind-config.yaml b/examples/rollout/kind-config.yaml new file mode 100644 index 000000000..59e43fae6 --- /dev/null +++ b/examples/rollout/kind-config.yaml @@ -0,0 +1,30 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# A cluster big enough to see compartments at work: one control-plane node and +# six workers. +# +# kind create cluster --name nodewright-examples --config examples/rollout/kind-config.yaml + +kind: Cluster +apiVersion: kind.x-k8s.io/v1alpha4 +nodes: + - role: control-plane + - role: worker + - role: worker + - role: worker + - role: worker + - role: worker + - role: worker diff --git a/examples/rollout/nodewright.yaml b/examples/rollout/nodewright.yaml new file mode 100644 index 000000000..aca364191 --- /dev/null +++ b/examples/rollout/nodewright.yaml @@ -0,0 +1,49 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Rolls a package out under the `staged-rollout` DeploymentPolicy. Apply +# deployment-policy.yaml first: the webhook rejects a reference to a policy that +# does not exist. +# +# Opt nodes in with: +# kubectl label node nodewright.nvidia.com/example=rollout +# then sort them into compartments with the example-tier label; see README.md. + +apiVersion: nodewright.nvidia.com/v1alpha1 +kind: NodeWright +metadata: + name: rollout-demo +spec: + deploymentPolicy: staged-rollout + nodeSelectors: + matchLabels: + nodewright.nvidia.com/example: rollout + packages: + stamp: + version: 1.0.0 + image: ghcr.io/nvidia/nodewright-packages/shellscript + configMap: + apply.sh: |- + #!/bin/bash + mkdir -p /etc/nodewright-example/rollout + if [ -e /etc/nodewright-example/rollout/fail ]; then + echo "$(hostname): /etc/nodewright-example/rollout/fail exists, failing on purpose" + exit 1 + fi + sleep 20 + date -u > /etc/nodewright-example/rollout/stamp + apply_check.sh: |- + #!/bin/bash + test -s /etc/nodewright-example/rollout/stamp diff --git a/examples/runtime-required/README.md b/examples/runtime-required/README.md new file mode 100644 index 000000000..2dccfd1aa --- /dev/null +++ b/examples/runtime-required/README.md @@ -0,0 +1,82 @@ +# Example: keep workloads off a node until it is ready + +Some node setup has to finish before anything else runs there: a driver, a +security agent, a kernel setting. `runtimeRequired: true` makes that a rule. The +node carries a taint that blocks ordinary pods, and the operator lifts it only +once the NodeWright is complete **on that node**. + +Package pods tolerate the taint automatically. Your workloads do not, which is the point. + +## How it works + +1. The node has the runtime-required taint, `nodewright.nvidia.com=runtime-required:NoSchedule` + by default. +2. A NodeWright with `runtimeRequired: true` targets the node and runs its packages. +3. When they finish on that node, the operator removes the taint. +4. Pods that were waiting get scheduled. + +The taint is per node. A slow node does not hold back a healthy one. + +## Run it + +Pick one node, label it, and taint it by hand. Pre-tainting is how you would do it +in production, by having the node join the cluster already tainted: + +```bash +NODE= +kubectl label node "$NODE" nodewright.nvidia.com/example=runtime-required +kubectl taint node "$NODE" nodewright.nvidia.com=runtime-required:NoSchedule +``` + +Start a workload pinned to that node. It has no toleration: + +```bash +kubectl apply -f examples/runtime-required/workload.yaml +kubectl get pod runtime-required-demo +``` + +It is `Pending`. `kubectl describe pod runtime-required-demo` shows why: the node +has a taint the pod does not tolerate. + +Now apply the NodeWright: + +```bash +kubectl apply -f examples/runtime-required/nodewright.yaml +kubectl get nodewright runtime-required-demo -w +``` + +The package takes about 45 seconds. When it completes, the taint comes off and the +pod starts: + +```bash +kubectl get node "$NODE" -o jsonpath='{.spec.taints}' ; echo +kubectl get pod runtime-required-demo +``` + +## What it will not do + +- **It will not taint a node for you** unless you also set `autoTaintNewNodes: true`. + Without a taint on the node, `runtimeRequired` gates nothing. +- **`autoTaintNewNodes` is a fallback, not a replacement for pre-tainting.** There is a + short window between a node joining and the operator tainting it. It also treats a + node as new only if no NodeWright has ever touched it, and `kubectl nodewright reset` + does not make it new again. +- **A broken package leaves the node tainted.** Test packages on a few nodes before + setting `runtimeRequired` across a fleet. +- **Do not combine it with `sequencing: all`.** One unhealthy node would hold the + whole fleet tainted. See [Strict Ordering](../../docs/architecture/ordering.md#caution-deadlock-risks). + +## Clean up + +```bash +kubectl delete -f examples/runtime-required/workload.yaml +kubectl delete -f examples/runtime-required/nodewright.yaml +kubectl label node "$NODE" nodewright.nvidia.com/example- +``` + +## Related + +- [Runtime Required](../../docs/user-guide/runtime-required.md): auto-tainting, + the taint key rename from `skyhook.nvidia.com`, and `runtimeRequiredCordonAfter` +- [Taints](../../docs/user-guide/taints.md): how package pods tolerate taints, and + what `blocked` means diff --git a/examples/runtime-required/nodewright.yaml b/examples/runtime-required/nodewright.yaml new file mode 100644 index 000000000..d9c061104 --- /dev/null +++ b/examples/runtime-required/nodewright.yaml @@ -0,0 +1,55 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Keeps ordinary workloads off a node until this NodeWright has finished on it. +# +# The node must already carry the runtime-required taint; README.md applies it by +# hand. The operator removes the taint once every runtimeRequired NodeWright +# targeting the node is complete on that node. +# +# Opt a node in with: +# kubectl label node nodewright.nvidia.com/example=runtime-required + +apiVersion: nodewright.nvidia.com/v1alpha1 +kind: NodeWright +metadata: + name: runtime-required-demo +spec: + nodeSelectors: + matchLabels: + nodewright.nvidia.com/example: runtime-required + runtimeRequired: true + + # Taints new nodes itself instead of expecting them to join already tainted. + # It is a fallback: there is a short window before the taint lands, and it + # never re-taints a node NodeWright has touched. Pre-tainting at provisioning + # is the dependable way. Leave it off for this walkthrough. + # autoTaintNewNodes: true + + packages: + node-readiness: + version: 1.0.0 + image: ghcr.io/nvidia/nodewright-packages/shellscript + configMap: + apply.sh: |- + #!/bin/bash + # Stand-in for work that must finish before workloads start, such as + # installing a driver. The sleep leaves time to watch the node gated. + sleep 45 + mkdir -p /etc/nodewright-example/runtime-required + date -u > /etc/nodewright-example/runtime-required/ready + apply_check.sh: |- + #!/bin/bash + test -s /etc/nodewright-example/runtime-required/ready diff --git a/examples/runtime-required/workload.yaml b/examples/runtime-required/workload.yaml new file mode 100644 index 000000000..75d535d30 --- /dev/null +++ b/examples/runtime-required/workload.yaml @@ -0,0 +1,31 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# An ordinary workload with no tolerations, pinned to the example node. It stays +# Pending while the runtime-required taint is on the node. + +apiVersion: v1 +kind: Pod +metadata: + name: runtime-required-demo + namespace: default +spec: + nodeSelector: + nodewright.nvidia.com/example: runtime-required + restartPolicy: Never + containers: + - name: workload + image: busybox:1.38.0 + command: ["sh", "-c", "echo started on $(hostname); sleep infinity"] diff --git a/examples/secrets/README.md b/examples/secrets/README.md new file mode 100644 index 000000000..baf2404de --- /dev/null +++ b/examples/secrets/README.md @@ -0,0 +1,68 @@ +# Example: giving a package a secret + +NodeWright does not fetch or inject secrets. It hands a package's `env` straight +to the package pod, so you use the Kubernetes mechanism you already have: put the +secret in a `Secret` and point an environment variable at it with `secretKeyRef`. +The package's scripts then read the variable. + +Never put a secret in a package's `configMap`. That map is copied verbatim into a +ConfigMap, which is readable by anyone with ConfigMap access, and it lands in the +NodeWright resource itself. + +## Run it + +Package pods run in the **operator's namespace**, so the Secret has to be there. +That is `nodewright` for a new install (older installs use `skyhook`): + +```bash +kubectl create secret generic nodewright-example \ + --namespace nodewright \ + --from-literal=token=REPLACE_ME +``` + +Replace `REPLACE_ME`, though any value works for the demo. Then: + +```bash +NODE= +kubectl label node "$NODE" nodewright.nvidia.com/example=secrets +kubectl apply -f examples/secrets/nodewright.yaml +kubectl get nodewright secrets-demo -w +``` + +Read what the script saw: + +```bash +kubectl nodewright package logs use-secret --nodewright secrets-demo --stage apply +``` + +```text +received a token of 11 characters +``` + +The script reports the length, not the value. Package logs are readable by anyone +who can read pods in the operator's namespace. + +## What to know + +- **The Secret and the package pod share a namespace**, so create it in the + operator's namespace, not `default`. +- **If the Secret or key is missing**, Kubernetes cannot start the package's + containers, so the package never finishes. The reason is in the events from + `kubectl describe pod -n nodewright `. +- **The value is read when the pod starts.** Rotating the Secret does not change a + package that already completed. Use `kubectl nodewright package rerun` to run it + again. +- `env` takes standard `EnvVar` entries, so to pass several keys, add one entry per + key. + +## Clean up + +```bash +kubectl delete -f examples/secrets/nodewright.yaml +kubectl delete secret nodewright-example --namespace nodewright +kubectl label node "$NODE" nodewright.nvidia.com/example- +``` + +## Related + +- [Providing Secrets to Packages](../../docs/user-guide/providing-secrets.md) diff --git a/examples/secrets/nodewright.yaml b/examples/secrets/nodewright.yaml new file mode 100644 index 000000000..b873d39fe --- /dev/null +++ b/examples/secrets/nodewright.yaml @@ -0,0 +1,54 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Hands a Kubernetes Secret to a package as an environment variable. +# +# The Secret is not part of this manifest. Package pods run in the operator's +# namespace, so the Secret must be created there first; see README.md. +# +# Opt a node in with: +# kubectl label node nodewright.nvidia.com/example=secrets + +apiVersion: nodewright.nvidia.com/v1alpha1 +kind: NodeWright +metadata: + name: secrets-demo +spec: + nodeSelectors: + matchLabels: + nodewright.nvidia.com/example: secrets + packages: + use-secret: + version: 1.0.0 + image: ghcr.io/nvidia/nodewright-packages/shellscript + env: + - name: EXAMPLE_TOKEN + valueFrom: + secretKeyRef: + name: nodewright-example + key: token + configMap: + apply.sh: |- + #!/bin/bash + # Never print the value: package logs are readable by anyone who can + # read pod logs in the operator's namespace. + if [ -z "${EXAMPLE_TOKEN}" ]; then + echo "EXAMPLE_TOKEN is empty" + exit 1 + fi + echo "received a token of ${#EXAMPLE_TOKEN} characters" + apply_check.sh: |- + #!/bin/bash + test -n "${EXAMPLE_TOKEN}" diff --git a/examples/simple/scr.yaml b/examples/simple/scr.yaml index 2fcc22e97..1a447c271 100644 --- a/examples/simple/scr.yaml +++ b/examples/simple/scr.yaml @@ -37,14 +37,26 @@ spec: nodewright.nvidia.com/quickstart: "true" packages: hello-world: - version: 1.1.0 - image: ghcr.io/nvidia/skyhook-packages/shellscript + version: 1.1.2 + image: ghcr.io/nvidia/nodewright-packages/shellscript configMap: - config.yaml: |- + # Stages run in order: apply first, then config. Each stage runs its + # script, then its _check script, and the stage is only done once the + # check passes. + apply.sh: |- + #!/bin/bash + sleep 1 + echo "Hello, apply!" + apply_check.sh: |- + #!/bin/bash + sleep 1 + echo "Hello, apply check!" + # Runs only after apply and apply_check have succeeded. + config.sh: |- #!/bin/bash sleep 1 echo "Hello, config!" - config_check.yaml: |- + config_check.sh: |- #!/bin/bash sleep 1 echo "Hello, config check!" diff --git a/examples/uninstall/README.md b/examples/uninstall/README.md new file mode 100644 index 000000000..00e83f370 --- /dev/null +++ b/examples/uninstall/README.md @@ -0,0 +1,75 @@ +# Example: explicit uninstall + +Deleting a NodeWright, or relabelling a node out of its selector, leaves the +host as the packages left it. Reversing a change is a separate, deliberate step: +uninstall. + +A package opts in with `uninstall.enabled: true`, which declares that it ships +`uninstall` and `uninstall_check` steps. Setting `uninstall.apply: true` then runs +them on every target node. + +This example's package writes a marker file on the host and removes it again. + +## Run it + +```bash +NODE= +kubectl label node "$NODE" nodewright.nvidia.com/example=uninstall +kubectl apply -f examples/uninstall/nodewright.yaml +kubectl get nodewright uninstall-demo -w +``` + +Once it is `complete`, the node's record of the package shows it installed: + +```bash +kubectl get node "$NODE" \ + -o jsonpath='{.metadata.annotations.nodewright\.nvidia\.com/nodeState_uninstall-demo}' +``` + +The file is on the host. [Check from a debug pod](../README.md#reading-files-on-a-node): +`chroot /host cat /etc/nodewright-example/uninstall/marker`. + +## Uninstall it + +Flip `apply` to `true`: + +```bash +kubectl patch nodewright uninstall-demo --type=merge \ + -p '{"spec":{"packages":{"marker":{"uninstall":{"apply":true}}}}}' +``` + +The operator runs `uninstall.sh`, then `uninstall_check.sh`, which fails unless the +file is gone. When the check passes, the package disappears from the node's +`nodeState_uninstall-demo` annotation. **Absent means uninstalled**: there is no +"uninstalled" state to look for. + +```bash +kubectl nodewright package status marker --nodewright uninstall-demo +``` + +Now it is safe to remove the package from the spec. The webhook would have +rejected that before the uninstall finished, which is what stops you from +orphaning files on a host. + +## Things to try + +- **Cancel.** Patch `apply` back to `false` while the uninstall is running. The + package installs again. (A package that declares an `interrupt` can no longer be + cancelled once its uninstall interrupt has fired.) +- **Uninstall on delete.** Reinstall, then `kubectl delete` the NodeWright. A + finalizer runs the uninstall on every node before the resource goes away. +- **Downgrade.** Lower `version` while `apply` is `false`. The webhook rejects it: + to downgrade a package with `uninstall.enabled`, uninstall it first. + +## Clean up + +```bash +kubectl delete -f examples/uninstall/nodewright.yaml +kubectl label node "$NODE" nodewright.nvidia.com/example- +``` + +## Related + +- [Explicit Uninstall](../../docs/user-guide/uninstall.md): every workflow, the + deletion edge cases, and how uninstall interacts with `dependsOn` +- [Lifecycle of a NodeWright](../../docs/architecture/lifecycle.md): the uninstall lane diff --git a/examples/uninstall/nodewright.yaml b/examples/uninstall/nodewright.yaml new file mode 100644 index 000000000..4d9678604 --- /dev/null +++ b/examples/uninstall/nodewright.yaml @@ -0,0 +1,54 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# A package that supports explicit uninstall. +# +# `uninstall.enabled: true` declares that the package ships uninstall.sh and +# uninstall_check.sh. Nothing is uninstalled until `uninstall.apply` is flipped +# to true; see README.md for the walkthrough. +# +# Opt a node in with: +# kubectl label node nodewright.nvidia.com/example=uninstall + +apiVersion: nodewright.nvidia.com/v1alpha1 +kind: NodeWright +metadata: + name: uninstall-demo +spec: + nodeSelectors: + matchLabels: + nodewright.nvidia.com/example: uninstall + packages: + marker: + version: 1.0.0 + image: ghcr.io/nvidia/nodewright-packages/shellscript + uninstall: + enabled: true + apply: false + configMap: + apply.sh: |- + #!/bin/bash + mkdir -p /etc/nodewright-example/uninstall + echo "managed by nodewright" > /etc/nodewright-example/uninstall/marker + apply_check.sh: |- + #!/bin/bash + grep -q "managed by nodewright" /etc/nodewright-example/uninstall/marker + uninstall.sh: |- + #!/bin/bash + rm -f /etc/nodewright-example/uninstall/marker + rmdir /etc/nodewright-example/uninstall 2>/dev/null || true + uninstall_check.sh: |- + #!/bin/bash + test ! -e /etc/nodewright-example/uninstall/marker From 62e8c41d31985cae1fb3b2a6f91b31c4d5be9996 Mon Sep 17 00:00:00 2001 From: James Lavoy Date: Mon, 26 Oct 2026 09:14:45 -0600 Subject: [PATCH 2/3] docs(examples): address review feedback Signed-off-by: James Lavoy --- examples/rollout/README.md | 37 +++++++++++++++---------- examples/rollout/deployment-policy.yaml | 9 ++++-- examples/runtime-required/README.md | 5 ++++ examples/secrets/README.md | 32 ++++++++++++++------- examples/secrets/nodewright.yaml | 2 +- examples/uninstall/README.md | 5 ++-- 6 files changed, 61 insertions(+), 29 deletions(-) diff --git a/examples/rollout/README.md b/examples/rollout/README.md index c07cd24c7..f4baa4ac7 100644 --- a/examples/rollout/README.md +++ b/examples/rollout/README.md @@ -9,9 +9,9 @@ This example sorts six workers into three groups: | Group | Nodes | Rolls out | Stops when | |---|---|---|---| -| `canary` | 1 | one at a time | the first batch fails | -| `production` | 3 | 1 node, then 2, growing by one per good batch | two batches in a row fail | -| default | 2 | two at a time | the first batch fails | +| `canary` | 2 | one at a time | a batch fails, so the second node never starts | +| `production` | 3 | 1 node, then 2, growing by one per good batch | two batches in a row fail, with a node still waiting | +| default | 1 | up to two at a time | (it has a single node, so there is nothing left to stop) | Nodes that match no compartment fall into the policy's `default`. @@ -35,11 +35,13 @@ WORKERS=($(kubectl get nodes -l '!node-role.kubernetes.io/control-plane' -o name for n in "${WORKERS[@]}"; do kubectl label node "$n" nodewright.nvidia.com/example=rollout done -kubectl label node "${WORKERS[0]}" nodewright.nvidia.com/example-tier=canary -for n in "${WORKERS[@]:1:3}"; do +for n in "${WORKERS[@]:0:2}"; do + kubectl label node "$n" nodewright.nvidia.com/example-tier=canary +done +for n in "${WORKERS[@]:2:3}"; do kubectl label node "$n" nodewright.nvidia.com/example-tier=production done -# the last two carry no tier label and fall into `default` +# the last one carries no tier label and falls into `default` ``` Create the policy before the NodeWright, because the webhook rejects a reference @@ -64,8 +66,8 @@ time. ## See a compartment stop The package fails on any node that has `/etc/nodewright-example/rollout/fail`. -Make the canary node fail, then run the rollout again. From a -[debug pod on the canary node](../README.md#reading-files-on-a-node), run +Which canary node goes first is not something to rely on, so make **both** canary +nodes fail. From a [debug pod on each canary node](../README.md#reading-files-on-a-node), run `chroot /host sh -c 'mkdir -p /etc/nodewright-example/rollout && touch /etc/nodewright-example/rollout/fail'`. A completed package is not re-run on its own, so reset the NodeWright to send it through again: @@ -74,14 +76,15 @@ through again: kubectl nodewright reset rollout-demo --confirm ``` -The canary node retries, then reads `erroring`. Its batch counts as failed, and -with `failureThreshold: 1` the **canary compartment stops**. The `production` and -default compartments keep going: stopping applies to the compartment that failed, -not the whole NodeWright. +The first canary node retries, then reads `erroring`. Its batch failed with 1 of 2 +canary nodes processed, which is 50%. That is below `safetyLimit: 100`, so +`failureThreshold: 1` is enforced and the **canary compartment stops**: the second +canary node never starts. The `production` and default compartments keep going. +Stopping applies to the compartment that failed, not the whole NodeWright. A stopped compartment stays stopped until its batch state is reset. After -removing the marker file, `reset` clears the failed node and the batch state -together, and the rollout starts over: +removing the marker files from both canary nodes, `reset` clears the failed node and +the batch state together, and the rollout starts over: ```bash kubectl nodewright reset rollout-demo --confirm @@ -93,6 +96,10 @@ kubectl nodewright reset rollout-demo --confirm nodes run at once.** The strategy sets the batch size. To cap concurrency, use `fixed` with `initialBatch: N`. The budget still breaks ties for a node that matches several compartments. +- **A compartment cannot stop on its last batch.** Stopping needs progress, counting + failed nodes as processed, to be below `safetyLimit`. When the final batch fails, + progress reaches 100% and there is nothing left to hold back. That is why the canary + group has two nodes: with one, a failure could never stop anything. - **`safetyLimit` defaults to 50.** Past half of a compartment, the rollout commits to finishing and stops enforcing `failureThreshold`. These manifests set it to 100 so a failure stops a rollout at any point. @@ -105,6 +112,8 @@ kubectl delete -f examples/rollout/deployment-policy.yaml for n in "${WORKERS[@]}"; do kubectl label node "$n" nodewright.nvidia.com/example- nodewright.nvidia.com/example-tier- done +# if you planted the failure marker, remove it from the canary nodes (debug pod): +# chroot /host rm -rf /etc/nodewright-example/rollout kind delete cluster --name nodewright-examples # if you made one ``` diff --git a/examples/rollout/deployment-policy.yaml b/examples/rollout/deployment-policy.yaml index 4e2bc36d6..7f4ff9a70 100644 --- a/examples/rollout/deployment-policy.yaml +++ b/examples/rollout/deployment-policy.yaml @@ -17,12 +17,17 @@ # with its own batch strategy; they roll out independently of one another. # # canary one node at a time, stop on the first failed batch -# production starts at one node and grows by one after each good batch -# default every node that matches no compartment: two at a time +# production starts at one node and grows by one after each good batch, +# stops after two failed batches in a row +# default every node that matches no compartment: up to two at a time # # `safetyLimit: 100` means failureThreshold is enforced for the whole rollout. # At the default of 50 the rollout commits to finishing once half the # compartment has been processed, whatever failureThreshold says. +# +# A compartment can only stop with nodes still waiting: a failure in its last +# batch leaves progress at 100%, which is not below the safety limit. A canary +# needs at least two nodes to be able to stop. apiVersion: nodewright.nvidia.com/v1alpha1 kind: DeploymentPolicy diff --git a/examples/runtime-required/README.md b/examples/runtime-required/README.md index 2dccfd1aa..80947f30c 100644 --- a/examples/runtime-required/README.md +++ b/examples/runtime-required/README.md @@ -72,8 +72,13 @@ kubectl get pod runtime-required-demo kubectl delete -f examples/runtime-required/workload.yaml kubectl delete -f examples/runtime-required/nodewright.yaml kubectl label node "$NODE" nodewright.nvidia.com/example- +kubectl taint node "$NODE" nodewright.nvidia.com=runtime-required:NoSchedule- ``` +The last command matters if you clean up before the package finished: with no +NodeWright left to remove it, the taint stays and the node stays unschedulable. +It reports `not found` if the operator already removed the taint, which is fine. + ## Related - [Runtime Required](../../docs/user-guide/runtime-required.md): auto-tainting, diff --git a/examples/secrets/README.md b/examples/secrets/README.md index baf2404de..e6aaeab6f 100644 --- a/examples/secrets/README.md +++ b/examples/secrets/README.md @@ -12,15 +12,26 @@ NodeWright resource itself. ## Run it Package pods run in the **operator's namespace**, so the Secret has to be there. -That is `nodewright` for a new install (older installs use `skyhook`): +Set it to wherever your operator is installed: `nodewright` for a new install, +`skyhook` for one that predates the rename. ```bash +NS=nodewright +``` + +Create the Secret without putting the value in the command, so it stays out of your +shell history and the process list. Type or paste a value (any value works for the +demo) when prompted; nothing is echoed: + +```bash +printf 'Token: '; read -rs TOKEN; echo kubectl create secret generic nodewright-example \ - --namespace nodewright \ - --from-literal=token=REPLACE_ME + --namespace "$NS" \ + --from-file=token=<(printf %s "$TOKEN") +unset TOKEN ``` -Replace `REPLACE_ME`, though any value works for the demo. Then: +Then: ```bash NODE= @@ -36,19 +47,20 @@ kubectl nodewright package logs use-secret --nodewright secrets-demo --stage app ``` ```text -received a token of 11 characters +EXAMPLE_TOKEN is set ``` -The script reports the length, not the value. Package logs are readable by anyone -who can read pods in the operator's namespace. +The script reports that the variable is set and nothing about its value, not even +its length. Package logs are readable by anyone who can read pods in the operator's +namespace. ## What to know - **The Secret and the package pod share a namespace**, so create it in the - operator's namespace, not `default`. + operator's namespace (`$NS`), not `default`. - **If the Secret or key is missing**, Kubernetes cannot start the package's containers, so the package never finishes. The reason is in the events from - `kubectl describe pod -n nodewright `. + `kubectl describe pod -n "$NS" `. - **The value is read when the pod starts.** Rotating the Secret does not change a package that already completed. Use `kubectl nodewright package rerun` to run it again. @@ -59,7 +71,7 @@ who can read pods in the operator's namespace. ```bash kubectl delete -f examples/secrets/nodewright.yaml -kubectl delete secret nodewright-example --namespace nodewright +kubectl delete secret nodewright-example --namespace "$NS" kubectl label node "$NODE" nodewright.nvidia.com/example- ``` diff --git a/examples/secrets/nodewright.yaml b/examples/secrets/nodewright.yaml index b873d39fe..1222644a5 100644 --- a/examples/secrets/nodewright.yaml +++ b/examples/secrets/nodewright.yaml @@ -48,7 +48,7 @@ spec: echo "EXAMPLE_TOKEN is empty" exit 1 fi - echo "received a token of ${#EXAMPLE_TOKEN} characters" + echo "EXAMPLE_TOKEN is set" apply_check.sh: |- #!/bin/bash test -n "${EXAMPLE_TOKEN}" diff --git a/examples/uninstall/README.md b/examples/uninstall/README.md index 00e83f370..883e17cc0 100644 --- a/examples/uninstall/README.md +++ b/examples/uninstall/README.md @@ -1,8 +1,9 @@ # Example: explicit uninstall Deleting a NodeWright, or relabelling a node out of its selector, leaves the -host as the packages left it. Reversing a change is a separate, deliberate step: -uninstall. +host as the packages left it, unless the package has opted in to uninstall (below), +in which case deleting the NodeWright runs the uninstall first. Reversing a change +is otherwise a separate, deliberate step: uninstall. A package opts in with `uninstall.enabled: true`, which declares that it ships `uninstall` and `uninstall_check` steps. Setting `uninstall.apply: true` then runs From 0af0f928912a3c8feea60f6f12b9b4f8a5b916f5 Mon Sep 17 00:00:00 2001 From: James Lavoy Date: Mon, 26 Oct 2026 09:37:47 -0600 Subject: [PATCH 3/3] docs(examples): address review feedback Signed-off-by: James Lavoy --- examples/rollout/README.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/examples/rollout/README.md b/examples/rollout/README.md index f4baa4ac7..825b3cc1c 100644 --- a/examples/rollout/README.md +++ b/examples/rollout/README.md @@ -17,7 +17,8 @@ Nodes that match no compartment fall into the policy's `default`. ## Prerequisites -Six or more worker nodes. To make a throwaway cluster: +At least six worker nodes; the walkthrough uses exactly six and ignores any +others. To make a throwaway cluster: ```bash kind create cluster --name nodewright-examples --config examples/rollout/kind-config.yaml @@ -30,7 +31,7 @@ and [install NodeWright](../../docs/getting-started/quickstart.md#1-install-the- Opt every worker in, then sort them into compartments: ```bash -WORKERS=($(kubectl get nodes -l '!node-role.kubernetes.io/control-plane' -o name | cut -d/ -f2)) +WORKERS=($(kubectl get nodes -l '!node-role.kubernetes.io/control-plane' -o name | cut -d/ -f2 | head -6)) for n in "${WORKERS[@]}"; do kubectl label node "$n" nodewright.nvidia.com/example=rollout