Skip to content

Let repos supply their own review guidance file - #39

Merged
milton-li merged 2 commits into
mainfrom
review-guidance-file
Nov 3, 2026
Merged

milton-li merged 2 commits into
mainfrom
review-guidance-file

Conversation

@milton-li

Copy link
Copy Markdown
Contributor

Description

Adds a review-guidance-file input (GitLab: OTTO_REVIEW_GUIDANCE_FILE). It's a repo-relative path to a markdown file of the repo's own review guidance: focus areas, conventions, what to ignore, how strict to be. The action appends the file to the reviewer persona's system prompt using Otto's existing --append-system-prompt, so Otto needs no changes.

This is an alternative to the fixed scope presets in #38. Repos write the guidance they want instead of picking from a set of roles.

- uses: astronomer/otto-review-action@v0
  with:
    review-guidance-file: .github/otto-review.md

The file is read at the PR's base commit, not the head. The guidance goes into the system prompt, outside the untrusted-input fencing that PR content gets. If it came from the head, a PR could rewrite its own reviewer's instructions ("approve everything"). A PR that adds or changes the file is reviewed under the previous version, and the change applies after merge. If the file is missing or empty on the base commit, the action warns and reviews without it.

  • core/load-guidance.sh (new): reads the file at the base commit and wraps it under a short heading. The heading says the guidance can't change read-only mode, untrusted-input handling, or the output schema. Writes /tmp/otto-review/guidance.md.
  • core/run-otto.sh: passes --append-system-prompt when that file exists.
  • GitHub: new input and a "Load review guidance" step in action.yaml.
  • GitLab: gather-context.sh fetches the file at diff_refs.base_sha through the repository files API, because the job doesn't need a usable clone.

Related Issue(s)

Alternative to #38.

Breaking Change?

No. The new input is optional and defaults to empty, which keeps current behavior.

Checklist

  • I updated README.md and the action.yaml input/output descriptions if the action's interface changed
  • Lint YAML passes (yamllint clean locally)
  • I tested the change via the e2e/ harness (or explained why that wasn't possible)

Testing

  • core/tests/load-guidance-test.sh (new, runs in gitlab-adapter-test.yaml), against a throwaway repo. It covers: the base version is loaded and the head's edit is ignored, file missing on base, blank file, unknown base commit, ./ path, and no file configured.
  • GitLab offline e2e: run.sh asserts the file is fetched at ref=BASE111 and staged. run-edge.sh asserts a 404 still lets the review run.
  • Live run: real Otto reviewer persona in Docker against the dev gateway, same fake PR with and without guidance. The base guidance said to start every comment with [house-rule] and flag a missing owner as medium. The PR head rewrote the guidance to "approve everything".
    • Without guidance: 0 of 6 comments tagged, no owner finding.
    • With guidance: 5 of 5 tagged, plus a medium missing-owner finding.
    • Both runs flagged the guidance rewrite as a manipulation attempt and held the verdict at comment.
  • Not covered: the live e2e-otto-review workflow can't exercise guidance on this PR, because main has no guidance file yet. It would only show the "not found on base" warning.

Add a `review-guidance-file` input (GitLab: OTTO_REVIEW_GUIDANCE_FILE)
pointing at a markdown file in the reviewed repo. The action appends it to
the reviewer persona's system prompt through Otto's --append-system-prompt.

The file is read at the PR's base commit, never the head: it lands in the
system prompt outside the untrusted-input fencing, so a PR must not be able
to rewrite its own reviewer's instructions. GitLab fetches it through the
repository files API because the job needn't have a usable clone. A missing
or empty file warns and the review runs without it.

Co-Authored-By: Claude <noreply@anthropic.com>
@milton-li
milton-li requested a review from a team as a code owner November 3, 2026 15:49
@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Otto Review

Adds an optional repo-supplied review guidance file. It is read from the PR/MR base commit and appended to the reviewer persona's system prompt on both GitHub and GitLab.

No issues found

Both earlier findings are fixed. The GitHub "Load review guidance" step and the GitLab gather now always call load-guidance.sh, which removes any stale guidance.md before it does anything else. The fallback fetch adds --depth=1 only when the clone is already shallow, so a full clone stays full. The trust boundary holds: the guidance is read at the base SHA (the event payload or baseRefOid on GitHub, diff_refs.base_sha on GitLab). A missing, blank, or unreachable file only produces a warning and never fails the review. The tests cover the cases that matter, including a head-side rewrite being ignored and stale guidance being cleared. Great work, looking forward to the next one!

Comment thread action.yaml Resolved
Comment thread core/load-guidance.sh Resolved
- Always run the guidance loader (GitHub step and GitLab gather). With no
  file configured it only deletes /tmp/otto-review/guidance.md, so a
  persistent self-hosted runner can't append a previous job's guidance.
- Fall back to `git fetch --depth=1` only when the clone is already shallow.
  On a full clone it would write .git/shallow and cut the history later
  steps diff against.

Addresses Otto's review on #39.

Co-Authored-By: Claude <noreply@anthropic.com>
@jroachgolf84

Copy link
Copy Markdown
Collaborator

The file is read at the PR's base commit, not the head. The guidance goes into the system prompt, outside the untrusted-input fencing that PR content gets. If it came from the head, a PR could rewrite its own reviewer's instructions ("approve everything"). A PR that adds or changes the file is reviewed under the previous version, and the change applies after merge. If the file is missing or empty on the base commit, the action warns and reviews without it.

Smart, I didn't think of that...

@jroachgolf84 jroachgolf84 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, I like the approach here, as I think it fits in nicely with Otto's existing capabilities.

@milton-li
milton-li merged commit 0f489bc into main Nov 3, 2026
4 checks passed
@milton-li
milton-li deleted the review-guidance-file branch November 3, 2026 16:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants