Skip to content

fix(queue): guard malformed execution-order annotations - #2952

Open
chmouel wants to merge 1 commit into
mainfrom
chmouel/fix-2945-annotation-panic
Open

fix(queue): guard malformed execution-order annotations#2952
chmouel wants to merge 1 commit into
mainfrom
chmouel/fix-2945-annotation-panic

Conversation

@chmouel

@chmouel chmouel commented Sep 1, 2026

Copy link
Copy Markdown
Member

📝 Description of the Change

If a PipelineRun ends up with a broken "which run is next" annotation
(empty, missing a slash, or with an extra slash in it), the watcher used
to crash instead of just skipping that broken entry. Because this
annotation gets read again every time the watcher starts up, one bad
entry would put the watcher into a permanent crash-restart-crash loop
for everyone, not just the one repository with the bad entry.

This change makes the watcher skip a broken entry instead of crashing
on it.

🔗 Linked GitHub Issue

Fixes #2945

JIRA: https://redhat.atlassian.net/browse/SRVKP-14113

🧪 Testing Strategy

  • Unit tests
  • Integration tests
  • End-to-end tests
  • Manual testing
  • Not Applicable

Added a test that feeds the parser an empty string, a value with no
slash, a value with an empty namespace, a value with an empty name, and
a value with an extra slash, plus one normal valid entry, and checks
none of them panic and the valid one is still parsed correctly.

🤖 AI Assistance

  • I have not used any AI assistance for this PR.
  • I have used AI assistance for this PR.

This PR was written with AI assistance (Claude Sonnet 5),
following a post-merge review of PR #2890. The bug, the fix, and the
test were all verified by hand: I reproduced the crash against the
merged code before writing the fix, and confirmed the fix and test pass.

✅ Submitter Checklist

  • 📝 My commit messages are clear, informative, and follow the project's How to write a git commit message guide. The Gitlint linter ensures in CI it's properly validated
  • ✨ I have ensured my commit message prefix (e.g., fix:, feat:) matches the "Type of Change" I selected above.
  • ♽ I have run make test and make lint locally to check for and fix any
    issues. For an efficient workflow, I have considered installing
    pre-commit and running pre-commit install to
    automate these checks.
  • 📖 I have added or updated documentation for any user-facing changes.
  • 🧪 I have added sufficient unit tests for my code changes.
  • 🎁 I have added end-to-end tests where feasible. See README for more details.
  • 🔎 I have addressed any CI test flakiness or provided a clear reason to bypass it.
  • If adding a provider feature, I have filled in the following and updated the provider documentation:
    • GitHub App
    • GitHub Webhook
    • Gitea/Forgejo
    • GitLab
    • Bitbucket Cloud
    • Bitbucket Data Center

Copilot AI lite review requested due to automatic review settings September 1, 2026 10:21
@codecov

codecov Bot commented Sep 1, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.44444% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 80.78%. Comparing base (aeed7ee) to head (a9c7fc9).

Files with missing lines Patch % Lines
pkg/reconciler/queue_pipelineruns.go 66.66% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2952      +/-   ##
==========================================
+ Coverage   80.66%   80.78%   +0.11%     
==========================================
  Files         164      164              
  Lines       13910    13911       +1     
==========================================
+ Hits        11221    11238      +17     
+ Misses       1967     1955      -12     
+ Partials      722      718       -4     
Flag Coverage Δ
unit-tests 80.78% <94.44%> (+0.11%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This pull request hardens queue startup/rebuild logic by preventing the watcher from panicking when it encounters malformed pipelinesascode.tekton.dev/execution-order annotation entries, avoiding a cluster-wide crash-loop scenario.

Changes:

  • Guard parsing of namespace/name execution-order entries using strings.Cut and skip malformed values instead of indexing blindly.
  • Add a unit test covering multiple malformed execution-order entries mixed with a valid one to ensure no panic and correct filtering.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
pkg/queue/queue_manager.go Adds validation around parsing namespace/name execution-order items to prevent panics on malformed entries.
pkg/queue/queue_manager_test.go Adds a regression test ensuring malformed execution-order entries are skipped without panicking.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread pkg/queue/queue_manager.go Outdated
@pipelines-as-code

pipelines-as-code Bot commented Sep 1, 2026

Copy link
Copy Markdown

Paco Review 🔍

The PR fixes a panic risk where malformed "namespace/name" queue keys (e.g., from a user-editable execution-order annotation) were indexed into via strings.Split without validating the split length, which could cause an index-out-of-range panic in the queue manager and reconciler. It introduces a shared SplitPrKey helper that safely parses and validates these keys, applies it at the FilterPipelineRunByState and queuePipelineRun call sites, and refactors finalizer.go's promotion logic to delegate to a new r.startNextPipelineRunInQueue call. Extensive new unit tests cover the malformed-key edge cases for both the queue package and the reconciler's finalize flow.

Review difficulty: 3/5 (Moderate) — The change spans queue and reconciler logic with panic-avoidance implications and includes a refactor whose new helper method definition is not visible in the diff, adding uncertainty despite solid test coverage.

1 new inline comment(s) found.

Reviewed commit: a9c7fc9

@pipelines-as-code pipelines-as-code Bot added paco/review-easy Paco review difficulty security-review Flagged as security-sensitive by Paco labels Sep 1, 2026

@pipelines-as-code pipelines-as-code Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Paco inline comments -- see the Paco Review summary comment for the overview.

Comment thread pkg/queue/queue_manager.go Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

@theakshaypant theakshaypant left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The original issue states

pkg/reconciler/finalizer.go:80 splits and indexes the same way and
should get the same guard.

Suggest adding the same guard there.

@theakshaypant theakshaypant left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also noticed the same call in reconciler.

@chmouel
chmouel force-pushed the chmouel/fix-2945-annotation-panic branch 2 times, most recently from 3400041 to 7c33337 Compare September 1, 2026 12:23
@chmouel

chmouel commented Sep 1, 2026

Copy link
Copy Markdown
Member Author

@theakshaypant thanks for catching that, pushed in the latest revision of this PR.

(desloped this comment since my agent answered for me while not asking to do it 🙃 )

@chmouel
chmouel force-pushed the chmouel/fix-2945-annotation-panic branch from 7c33337 to d88c1b3 Compare September 1, 2026 12:33
@chmouel
chmouel requested a balanced review from Copilot September 1, 2026 12:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Comment thread pkg/reconciler/finalizer.go Outdated
Split the "namespace/name" annotation with strings.Cut instead of
strings.Split and skip empty, slash-missing, or extra-slash entries
instead of indexing unconditionally. A malformed annotation
previously panicked FilterPipelineRunByState, which InitQueues calls
before workers start, crash-looping the watcher on restart.

Fixes #2945

Signed-off-by: Chmouel Boudjnah <chmouel@redhat.com>
Assisted-by: Claude
@chmouel
chmouel force-pushed the chmouel/fix-2945-annotation-panic branch from d88c1b3 to a9c7fc9 Compare September 1, 2026 15:22
@chmouel

chmouel commented Sep 1, 2026

Copy link
Copy Markdown
Member Author

/paco review

@pipelines-as-code pipelines-as-code Bot added paco/review-moderate Paco review difficulty and removed paco/review-easy Paco review difficulty labels Sep 1, 2026

@pipelines-as-code pipelines-as-code Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Paco inline comments -- see the Paco Review summary comment for the overview.

Comment thread pkg/reconciler/finalizer.go
next := r.qm.RemoveAndTakeItemFromQueue(repo, pr)
if next != "" {
key := strings.Split(next, "/")
pr, err := r.run.Clients.Tekton.TektonV1().PipelineRuns(key[0]).Get(ctx, key[1], metav1.GetOptions{})

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if this is deleted because of key[0] and key[1] then same is done in r.startNextPipelineRunInQueue as well so should we fix there as well?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ok-to-test paco/review-moderate Paco review difficulty security-review Flagged as security-sensitive by Paco

Projects

None yet

Development

Successfully merging this pull request may close these issues.

watcher crash-loops on a malformed execution-order annotation

4 participants