Skip to content

Upgrade Higress to 2.2.0 and add deployment health test#473

Merged
weilei0120 merged 7 commits intomainfrom
fix/andrewma/update_higress_version_in_bootstrap
Apr 23, 2026
Merged

Upgrade Higress to 2.2.0 and add deployment health test#473
weilei0120 merged 7 commits intomainfrom
fix/andrewma/update_higress_version_in_bootstrap

Conversation

@amd-ama10002-2
Copy link
Copy Markdown
Collaborator

Problem

When installing Higress 2.2.0 via Bootstrap/higress/higress.sh, the Higress controller fails to become ready. The higress-controller pod shows 0/2 Running and the higress-gateway pods get stuck in ContainerCreating.

Root cause: The script installs Gateway API CRDs v1.0.0, which only exposes BackendTLSPolicy as v1alpha2. Higress 2.2.0 expects v1.BackendTLSPolicy, which doesn't exist in that version.

You can confirm this by checking the controller logs:

kubectl logs -n higress-system -l app=higress-controller -c higress-core --tail=50

The key error is:

failed to list *v1.BackendTLSPolicy: the server could not find the requested resource (get backendtlspolicies.gateway.networking.k8s.io)

Summary

  • Upgrade the Higress Helm chart from 2.1.8 to 2.2.0 and switch the chart source from the private OCI registry (registry-1.docker.io/primussafe/higress) to the public Higress repo (higress.io/higress), since the private registry only hosts 2.1.8.
  • Upgrade Gateway API CRDs from v1.0.0 to v1.4.0 with --server-side apply. Higress 2.2.0 expects v1.BackendTLSPolicy, which only ships in Gateway API v1.4.0 (v1.0.0 only provides v1alpha2). Without this, the controller fails with failed to list *v1.BackendTLSPolicy.
  • Add a Kyverno Chainsaw infrastructure test for verifying the Higress deployment health.
  • Add Bootstrap/kubespray/ and Bootstrap/kubespray-venv/ to .gitignore (generated at runtime by bootstrap.sh).

What changed

File Change
Bootstrap/higress/higress.sh Helm chart source changed to higress.io/higress, version bumped to 2.2.0. Gateway API CRDs updated to v1.4.0 with --server-side.
Bootstrap/tests/higress/chainsaw-test.yaml New Chainsaw test that asserts: (1) higress-controller deployment has ready replicas, (2) at least one higress-gateway pod is Running, (3) ssh-gateway Gateway resource exists.
Bootstrap/tests/.chainsaw.yaml Shared Chainsaw configuration (timeouts, fail-fast).
Bootstrap/tests/README.md Documentation for running the infrastructure tests.
.gitignore Ignore Bootstrap/kubespray/ and Bootstrap/kubespray-venv/.

Why Gateway API v1.4.0?

The experimental channel (experimental-install.yaml) is required because Higress uses CRDs that the standard channel does not include (TCPRoute, BackendTLSPolicy, etc.). Version 1.4.0 is the minimum that serves v1.BackendTLSPolicy, which Higress 2.2.0 requires.

Test plan

Run the deployment script and then the Chainsaw test:

cd ~/Primus-SaFE/Bootstrap
bash higress/higress.sh

sudo nerdctl run --rm \
    -v ./tests/:/chainsaw/ \
    -v ${HOME}/.kube/:/etc/kubeconfig/ \
    -e KUBECONFIG=/etc/kubeconfig/config \
    --network=host \
    ghcr.io/kyverno/chainsaw \
    test /chainsaw --config /chainsaw/.chainsaw.yaml

Expected result: All 3 checks pass.

Known limitation

The controller logs show non-fatal RBAC warnings for experimental gateway.networking.x-k8s.io resources (xbackendtrafficpolicies, xlistenersets). These do not prevent the controller from reaching Ready state. The Higress Helm chart does not ship RBAC for these experimental CRDs; this can be addressed in a follow-up if needed.

Testing

Screenshot Installation works:

Details Screenshot 2026-04-07 204129

Screenshot of infra test passing -- pods are running.

Details Screenshot 2026-04-07 204154

Screenshot showing higress gateway is accessible:

Details image

Refactor the resource availability logic to enhance user experience. The flavor availability is now lazy-loaded when the dialog opens, and the GPU field is conditionally hidden based on the availability response. This change aims to streamline the dialog's performance and prevent unnecessary data loading.
Introduce a new README.md file detailing the end-to-end testing setup for Bootstrap components using Kyverno Chainsaw. The document includes prerequisites, instructions for running tests, test structure, and a table of available tests, enhancing the documentation for developers.
- Change the helm chart source for higress to higress.io and update the version to 2.2.0.
- Upgrade the Gateway API CRDs installation to version 1.4.0 with server-side apply for improved deployment.
- Add Bootstrap/kubespray/ and Bootstrap/kubespray-venv/ to .gitignore to prevent tracking of generated files in the Bootstrap infrastructure.
- Change the skipDelete option from true to false in the cleanup section to ensure resources are deleted after test execution.
@amd-ama10002-2 amd-ama10002-2 self-assigned this Apr 8, 2026
@amd-ama10002-2 amd-ama10002-2 marked this pull request as ready for review April 8, 2026 02:28
Copilot AI review requested due to automatic review settings April 8, 2026 02:28
@amd-ama10002-2 amd-ama10002-2 added the enhancement New feature or request label Apr 8, 2026
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the Bootstrap Higress installation to work with Higress 2.2.0 by upgrading the Helm chart source/version and upgrading the Kubernetes Gateway API CRDs to a version that includes v1.BackendTLSPolicy. It also introduces Kyverno Chainsaw-based infrastructure health checks for the Higress deployment and documents how to run them.

Changes:

  • Upgrade Higress Helm chart to 2.2.0 and switch chart source to higress.io/higress.
  • Upgrade Gateway API CRDs to v1.4.0 using server-side apply.
  • Add Chainsaw test/config + documentation for Bootstrap infrastructure tests; update .gitignore for generated Bootstrap directories.

Reviewed changes

Copilot reviewed 4 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
Bootstrap/higress/higress.sh Bumps Higress chart to 2.2.0 and updates Gateway API CRDs to v1.4.0 (server-side apply).
Bootstrap/tests/higress/chainsaw-test.yaml Adds a Chainsaw test to validate Higress controller readiness, gateway pods running, and presence of a Gateway resource.
Bootstrap/tests/.chainsaw.yaml Adds shared Chainsaw configuration (timeouts, fail-fast, cleanup behavior).
Bootstrap/tests/README.md Documents how to run the infrastructure tests and the test directory structure.
.gitignore Ignores runtime-generated Bootstrap/kubespray/ and Bootstrap/kubespray-venv/.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread Bootstrap/tests/higress/chainsaw-test.yaml Outdated
Comment thread Bootstrap/tests/.chainsaw.yaml
Comment thread Bootstrap/higress/higress.sh
Comment thread Bootstrap/tests/README.md Outdated
@amd-ama10002-2 amd-ama10002-2 marked this pull request as draft April 8, 2026 02:36
- Replace script-based check with an assert for the higress-gateway pod status in the chainsaw-test.yaml, ensuring the pod is Running in the specified namespace.
@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 8, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Clarify the nature of the tests by changing "unit tests" to "integration tests" in the README.md, providing a more accurate description of the testing framework used with Kyverno Chainsaw.
@amd-ama10002-2 amd-ama10002-2 marked this pull request as ready for review April 8, 2026 03:04
@weilei0120 weilei0120 merged commit ee6f14a into main Apr 23, 2026
29 checks passed
@weilei0120 weilei0120 deleted the fix/andrewma/update_higress_version_in_bootstrap branch April 23, 2026 09:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants