watcher and api-server horizontal scaling

[olegbet]

Changes

• added Watcher pods bucket-sharding scaling swithched on in Watcher's gRPC config ;
• deprecated blocking grpc.DialContext client connection function has been replaced with newer grpc.NewClient function. grpc.WithBlock has been removed and replaced with grpc.WithConnectParams(connectParams) to use new scaling parameters;
• added StatefulSet overlay tekton-results-watcher with 3 replicas to be used to scale Watcher pods instead of Deployment ;
• added Deployment overlay tekton-results-api scaled to 3 replicas;
• Headless service tekton-results-api-service-headless has been added with clusterIP: None for load balancing to be able to use DNS resolution;
• TLS certificate generation script added for the new headless service;
• Deployment scripts modified to deploy a new horizontally scaled configuration;
• Integration and functional tests added;
• Documentation updates about horizontal scaling;

Submitter Checklist

These are the criteria that every PR should meet, please check them off as you review them:

• Has Docs included if any changes are user facing
• Has Tests included if any functionality added or changed
• Tested your changes locally (if this is a code change)
• Follows the commit message standard
• Meets the Tekton contributor standards (including functionality, content, code)
• Has a kind label. You can add a comment on this PR that contains /kind <type>. Valid types are bug, cleanup, design, documentation, feature, flake, misc, question, tep
• Release notes block below has been updated with any user-facing changes (API changes, bug fixes, changes requiring upgrade notices or deprecation warnings)
• Release notes contain the string "action required" if the change requires additional action from users switching to the new release

Release Notes

Added a horizontal scaling functionality for Watcher / API Server high availability.
Added the high availability kustomize overlay to deploy the new horizontaly scaled configuration.
The Watcher and API server can now be deployed scaled to multiple pods and they will use DNS-based round-robin gRPC load balancing to manage workload properly.
By default the service will be deployed the single replica way but the process how to deploy the scaled configuration can be found in documentation.

Assisted-by: Claude

[tekton-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign khrm after the PR has been reviewed.
You can assign the PR to them by writing /assign @khrm in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[khrm]

Can you please remove reference to internal jira in the title?

[enarha]

Just to confirm, this is still WIP, right? If so, please indicate it on the PR e.g. prefix the PR subject with WIP: . This is also a significant change adding new functionality, so the Release Notes section of the PR description should include a short release note. The release notes are used when we create a new release to inform the users which changes are incorporated in that release.

[olegbet]

Can you please remove reference to internal jira in the title?

Should I create an upstream github issue for this feature?

[enarha]

Can you please remove reference to internal jira in the title?

Should I create an upstream github issue for this feature?

IMO that's not needed, just for the sake of having one. The GitHub issue is a way for the users and contributors to communicate issues and features requests. This one is almost done, so no really need for it.

[Copilot]

Pull request overview

Adds horizontal scaling for the Results API and watcher using headless services, gRPC round-robin balancing, and bucket-based watcher sharding.

Changes:

• Adds a three-replica API and watcher scaling component.
• Configures DNS-based gRPC balancing and TLS certificate generation.
• Adds deployment documentation and service-config validation.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
docs/install.md	Documents TLS setup for scaled deployments.
config/overlays/ha-local-db/kustomization.yaml	Adds an HA local-database overlay.
config/components/horizontal-scaling/watcher-statefulset.yaml	Defines the sharded watcher StatefulSet.
config/components/horizontal-scaling/watcher-headless-service.yaml	Adds watcher stable network identities.
config/components/horizontal-scaling/README.md	Documents scaling behavior and operation.
config/components/horizontal-scaling/kustomization.yaml	Composes and configures scaling resources.
config/components/horizontal-scaling/generate-tls-cert.sh	Generates SAN-enabled TLS certificates.
config/components/horizontal-scaling/api-service-headless.yaml	Exposes API pod endpoints through DNS.
cmd/watcher/main.go	Enables round-robin gRPC balancing.
cmd/watcher/main_test.go	Validates the default balancing configuration.

Suppressed comments (2)

config/components/horizontal-scaling/watcher-statefulset.yaml:84

• These stateful-election variables make Knative assign each ordinal an unopposed, fixed bucket rather than a lease-backed bucket. If any watcher pod is unavailable, no surviving replica takes its shard, so reconciliations for that shard stop; this does not provide the high availability promised by the component. Use standard bucket leader election for failover, or explicitly make the no-failover behavior a documented constraint.

            - name: STATEFUL_CONTROLLER_ORDINAL
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: STATEFUL_SERVICE_NAME

config/components/horizontal-scaling/README.md:192

• This compatibility claim is incorrect. logs-file patches only the base watcher Deployment, so the replacement StatefulSet never receives -logs_api; it also mounts one ReadWriteOnce PVC into the API Deployment that this component scales to three replicas. A successful Kustomize patch does not make the combined components functional.

- **Other components**: Components like `logs-file` that patch the watcher Deployment continue to work. The base Deployment is scaled to 0 replicas (not deleted), so patches apply without error.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[enarha]

Please also add ha-base-only overlay (the same as the one you added without the local-db component)

[olegbet]

Please also add ha-base-only overlay (the same as the one you added without the local-db component)

Done

[enarha]

Thank you for the PR. Few comments, nothing major.

Please squash the commits. There is no really need to keep piling them for every change, unless you really want to split the PR into multiple commits encapsulating different functionality. Even when you squash, the diff is visible in GitHub if someone is interested in what changed between two pushes.

Update the actual commit message as well as the PR description. See my comment below, the PR description should also include release note as well as kind label (it should be /kind feature in this case).

Update the doc with information how the metrics should be scraped when using that ha mode. Something like When using horizontal scaling, configure Prometheus to scrape individual pods (via PodMonitor or targeting the headless service) rather than the ClusterIP service. The ClusterIP only hits one random pod per scrape, giving an incomplete view. The headless services already expose the metrics port (9090), so they can be used as scrape targets directly..

[enarha]

Suggested change

	Copyright 2020 The Tekton Authors
	Copyright 2026 The Tekton Authors

[enarha]

I believe this is wrong. The bucket is left without an owner until the statefulset creates the replica with the respective ordinal.

[enarha]

Suggested change

	3. The new pod starts, cache syncs, and takes ownership of assigned buckets
	3. The new pod starts, cache syncs, and takes ownership of assigned bucket

[enarha]

Can you please share the reference to that source code/doc? Checking vendor/google.golang.org/grpc/internal/resolver/dns/dns_resolver.go I see MinResolutionInterval = 30 * time.Second and ResolvingTimeout = 30 * time.Second but nothing about retry timeout or 30 minutes.

[olegbet]

Seems like this statement is also not correct, I'll update the documentation.

[enarha]

this is still not updated

[olegbet]

updated in the last commit

[enarha]

I believe this is wrong, see my comment below.

[olegbet]

updated this part

[enarha]

This is a significant change of behavior, because of the removal of grpc.WithBlock() which is not supported with grpc.NewClient. The client is no longer blocking, instead it returns immediately and relies on lazy connection establishment. I'm not against the change, but it should be very well documented both as part of the commit message (both the actual commit message and PR description) and the release notes (part of the PR description).

[enarha]

I tested this manually and it created a bigger issue. The logs functionality is completely skipped because grpc.NewClient returns immediately and the DNS resolution is not complete yet. The message in the log Unable to inject logs client, logs will not be stored: rpc error: code = Unavailable desc = no children to pick from indicates that. I tested a fix which seems to be working:

diff --git a/cmd/watcher/main.go b/cmd/watcher/main.go
index 6b99887..624086e 100644
--- a/cmd/watcher/main.go
+++ b/cmd/watcher/main.go
@@ -191,6 +191,7 @@ func connectToAPIServer(apiAddr string, authMode string) (*grpc.ClientConn, erro
        opts := []grpc.DialOption{
                grpc.WithDefaultServiceConfig(defaultServiceConfig),
                grpc.WithConnectParams(connectParams),
+               grpc.WithDefaultCallOptions(grpc.WaitForReady(true)),
        }
        // Add in additional credentials to requests if desired.
        switch authMode {

this is also what https://github.com/grpc/grpc-go/blob/master/Documentation/anti-patterns.md recommends.

[enarha]

storage.googleapis.com is the old storage and will result the installation of very old version of Tekton Pipeline. test/e2e/01-install.sh has the actual updated host.

[enarha]

There is a lot of duplication (and some introduced errors) between this newly added script and the existing one test/e2e/01-install.sh. here is an Ai generated optimization idea. Please verify before using.

Script duplication: 01-install.sh, 01-install-ha.sh, generate-tls-cert.sh

The three scripts share most of their logic. 01-install-ha.sh is a near-complete copy of 01-install.sh with only a few differences:

1. Extra SAN in the TLS cert (api-service-headless)
2. Different kustomize overlay (kustomize-ha/ vs kustomize/)
3. Different pod wait strategy (label-based vs Deployment-based)

generate-tls-cert.sh duplicates the OpenSSL + LibreSSL cert generation a third time.

Suggestion

Collapse into a single 01-install.sh that accepts a mode parameter:

#!/bin/bash
set -e

MODE="${1:-standard}" # "standard" or "ha"

export KO_DOCKER_REPO=${KO_DOCKER_REPO:-"kind.local"}
export KIND_CLUSTER_NAME=${KIND_CLUSTER_NAME:-"tekton-results"}
export SA_TOKEN_PATH=${SA_TOKEN_PATH:-"/tmp/tekton-results/tokens"}
export SSL_CERT_PATH=${SSL_CERT_PATH:="/tmp/tekton-results/ssl"}

ROOT="$(git rev-parse --show-toplevel)"

# --- Install Tekton Pipelines (shared) ---
TEKTON_PIPELINE_CONFIG=${TEKTON_PIPELINE_CONFIG:-"https://infra.tekton.dev/tekton-releases/pipeline/latest/release.yaml"}
kubectl apply --filename "${TEKTON_PIPELINE_CONFIG}"
kubectl wait --for=condition=ready pod -l app=tekton-pipelines-controller -n tekton-pipelines --timeout=120s
kubectl wait --for=condition=ready pod -l app=tekton-pipelines-webhook -n tekton-pipelines --timeout=120s

# --- DB secret (shared) ---
kubectl create secret generic tekton-results-postgres \
    --namespace="tekton-pipelines" \
    --from-literal=POSTGRES_USER=postgres \
    --from-literal=POSTGRES_PASSWORD="$(openssl rand -base64 20)" || true

# --- TLS cert generation — delegate to the shared helper ---
export NAMESPACE="tekton-pipelines"
export OUTPUT_DIR="${SSL_CERT_PATH}"
export SECRET_NAME="tekton-results-tls"

if [ "$MODE" = "ha" ]; then
    # The generate-tls-cert.sh script already includes both the regular
    # and headless service names in the SAN.
    "${ROOT}/config/components/horizontal-scaling/generate-tls-cert.sh"
else
    # Standard mode: single SAN only.
    # Could extend generate-tls-cert.sh to accept EXTRA_SANS, or
    # keep the inline openssl call for the simple case.
    mkdir -p "${SSL_CERT_PATH}"
    altNames="DNS:tekton-results-api-service.tekton-pipelines.svc.cluster.local"
    if [ "${SSL_INCLUDE_LOCALHOST:-false}" = "true" ]; then
        altNames+=",DNS:localhost"
    fi
    openssl req -x509 \
        -newkey rsa:4096 \
        -keyout "${SSL_CERT_PATH}/key.pem" \
        -out "${SSL_CERT_PATH}/cert.pem" \
        -days 365 -nodes \
        -subj "/CN=tekton-results-api-service.tekton-pipelines.svc.cluster.local" \
        -addext "subjectAltName = ${altNames}"
    kubectl create secret tls -n tekton-pipelines tekton-results-tls \
        --cert="${SSL_CERT_PATH}/cert.pem" \
        --key="${SSL_CERT_PATH}/key.pem" || true
fi

# --- Deploy (mode-specific overlay) ---
if [ "$MODE" = "ha" ]; then
    KUSTOMIZE_DIR="${ROOT}/test/e2e/kustomize-ha"
else
    KUSTOMIZE_DIR="${ROOT}/test/e2e/kustomize"
fi
extra_ko_params="linux/$(go env GOARCH)"
kubectl kustomize "$KUSTOMIZE_DIR" | ko apply --platform="$extra_ko_params" -f -

# --- SA tokens (shared) ---
mkdir -p "${SA_TOKEN_PATH}"
for sa in all-namespaces-read-access single-namespace-read-access all-namespaces-admin-access all-namespaces-impersonate-access; do
    kubectl create token "$sa" > "${SA_TOKEN_PATH}/${sa}"
done

# --- Wait for pods (shared, label-based works for both modes) ---
kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=tekton-results-api -n tekton-pipelines --timeout=300s
kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=tekton-results-watcher -n tekton-pipelines --timeout=300s
kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=tekton-results-postgres -n tekton-pipelines --timeout=120s

Then update e2e.sh to call:

if [ "$1" == "--ha" ]; then
    ${REPO}/test/e2e/01-install.sh ha
else
    ${REPO}/test/e2e/01-install.sh
fi

To fully eliminate the third copy, generate-tls-cert.sh could be extended to support an EXTRA_SANS env var so it handles both the single-SAN and multi-SAN cases, making it usable as the sole cert generation entry point for all paths.

[olegbet]

Refactored the scripts

[enarha]

Run E2E tests: go test -v --tags=e2e,e2e_ha -run TestHorizontalScaling .
[ results] -> go test -v --tags=e2e,e2e_ha -run TestHorizontalScaling .

.

no Go files in /tmp/results
FAIL . [setup failed]
FAIL

that should be "Run E2E tests: go test -v --tags=e2e,e2e_ha -run TestHorizontalScaling ./..." or something stricter if you want to limit it to specific directory.

[olegbet]

Updated

[enarha]

/hold
something went wrong with that rebase

[olegbet]

/kind feature

[adityavshinde]

Missing liveness/readiness probes on the watcher StatefulSet:
We should define a livenessProbe or readinessProbe. Knative's 'shareadmin' already serves /heath and /readiness on port 8080 inside the watcher, but the StatefulSet spec never tells Kubernetes to check them. In the HA setup with bucket sharding, this means if one watcher replica gets stuck, K8s will keep reporting it as "Ready 1/1". That replica's bucket of TR/PR will silently stop being reconciled ( no alert or detection ).
IMO we can fix this by adding livenessProbe or readinessProbe to the container spec ( port 8080 also needs to be added )

[enarha]

While I agree with that, it's a pre-existing state and it applies to the Deployment as well (when we deploy as single replica). So IMO it makes sense to fix that in a separate PR which fixes the issue for both Deployment and StatefulSet. We can open a separate GitHub issue for it, so it's not forgotten.

[adityavshinde]

Yes, that makes sense to have a different PR for this.

[adityavshinde]

I have created an issue for the above: #1466

[adityavshinde]

The TEKTON_RESULTS_API_SERVICE env var hardcodes tekton-pipelines in the DNS name. The base Deployment has the same hardcoding, but it gets patched by the operator at deploy time. Since this StatefulSet is a new resource, the operator would also need to be updated to patch it for non-default namespaces.

[enarha]

Good call. I'll add to that, that the operator should not just patch, but ultimately manage that new Service object and the Route associated with it. Here we do not have that capability to change dynamically the type of deployment (single replicas vs scaleable deployments). The work on the operator side can be managed as a separate issue on the operator repo. We need that PR merged first and the next version of Results including that change to be referenced by the operator because of the change in the Watcher main file.

[adityavshinde]

PodDisruptionBudget (PDB) for HA components can be added:
This PR has 3 replica HA for both the watcher (StatefulSet) and API server (Deployment), but there is no PodDisruptionBudget to protect availability during voluntary disruptions like node drains or cluster upgrades.
Without a PDB, kubectl drain can evict all 3 watcher pods or all 3 API pods simultaneously if they happen to be co-located. With StatefulSet ordinals, evicted pods' buckets have no failover, so a drain that hits 2 pods means 2/3 of reconciliation stops until the pods are rescheduled.
This is not a blocker, but this can be a better fit for HA component.
We can add a policy under config/components/horizontal-scaling/

[olegbet]

PodDisruptionBudget (PDB) for HA components can be added:
This PR has 3 replica HA for both the watcher (StatefulSet) and API server (Deployment), but there is no PodDisruptionBudget to protect availability during voluntary disruptions like node drains or cluster upgrades.
Without a PDB, kubectl drain can evict all 3 watcher pods or all 3 API pods simultaneously if they happen to be co-located. With StatefulSet ordinals, evicted pods' buckets have no failover, so a drain that hits 2 pods means 2/3 of reconciliation stops until the pods are rescheduled.
This is not a blocker, but this can a better fit for HA component.
We can add a policy under config/components/horizontal-scaling/

Added PodDisruptionBudget to the watcher StatefulSet.

[adityavshinde]

I have reviewed this PR and the changes looks good to me. /lgtm

[enarha]

This is more of a implementation detail, while I see this doc as a customer focused documentation. IMO we can just drop that.

Suggested change

	### grpc.NewClient Migration
	The watcher uses `grpc.NewClient` (introduced in gRPC-Go 1.58) instead of the deprecated `grpc.DialContext`.
	**Key differences:**
	| `grpc.DialContext` (deprecated) | `grpc.NewClient` (current) |
	|--------------------------------|---------------------------|
	| Blocking by default | Non-blocking - returns immediately |
	| Accepts `context.Context` | No context parameter |
	| Supports `grpc.WithBlock()` | Does not support `grpc.WithBlock()` |
	| Connection established during dial | Connections established lazily on first RPC |

[olegbet]

dropped

[enarha]

Suggested change

	main $@
	main "$@"

[adityavshinde]

@enarha Thanks for pointing this out, I missed it. Yes it should be main "$@"

[olegbet]

I tried this locally and this works both ways but thanks anyways, I'll update it.

[olegbet]

fixed

[enarha]

I tried this locally and this works both ways but thanks anyways, I'll update it.

It works with the existing ha parameter, but that could break when the shell performs word splitting and glob expansion with other parameters we add to it. This is a shell correctness issue.

[enarha]

typo

Suggested change

	Note: Deprecated blocking grpc.DialContext client connection function has been replaced with newer grpc.NewClient function. grpc.WithBlock is uncompatible with the new scaling functionality and has been removed and replaced with grpc.WithConnectParams(connectParams) to use new scaling parameters;
	Note: Deprecated blocking grpc.DialContext client connection function has been replaced with newer grpc.NewClient function. grpc.WithBlock is incompatible with the new scaling functionality and has been removed and replaced with grpc.WithConnectParams(connectParams) to use new scaling parameters;

[olegbet]

fixed