-
Notifications
You must be signed in to change notification settings - Fork 5.4k
Add AI code review tools evaluation policy #8910
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,99 @@ | ||
| # AI Code Review Tools | ||
|
|
||
| The Kubernetes project may evaluate AI-powered code review tools on a | ||
| per-repo opt-in basis. This document describes the process for requesting, | ||
| evaluating, and deciding on the use of such tools. | ||
|
|
||
| ## Scope | ||
|
|
||
| This policy covers AI tools that automatically review pull requests, such as | ||
| CodeRabbit or GitHub Copilot code review. It does not cover other AI-powered | ||
| tooling such as CI/CD, security scanning, or code generation assistants. | ||
|
|
||
| ## Requesting a New Tool | ||
|
|
||
| A subproject lead or an approver listed in the repository's top-level [OWNERS] | ||
| file files an issue on [kubernetes/org]. The issue must: | ||
|
|
||
| - Identify the tool and link to its documentation | ||
| - Describe the use cases and what the subproject is trying to accomplish | ||
| - Explain why existing approved tools do not meet their needs | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is there a list of already-approved tools somewhere?
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. IIRC k8s repo admins can enable Copilot auto code reviews, but as @kannon92 mentioned in kubernetes/org#5930 (comment), it requires PR authors to have Copilot subscription first, or the maintainers need to manually request reviews from Copilot
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. OK. I'm just saying that if we have existing tools, we should link to a list somewhere, so that it'll save a lot of duplicate requests.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Speaking of this, if a tool is approved in one SIG repo/subproject, can we assume it's approved to use in other repos as well (may still need admin to configure it, but no additional approval is needed)?
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @janetkuo assuming there is not extra $$$'s involved perhaps?
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes agreed
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Added a blank section for approved tools |
||
| - List the specific repositories for the pilot | ||
| - Acknowledge the [AI guidance for pull requests] | ||
| - List the GitHub permissions and OAuth scopes the tool requests | ||
|
|
||
| ## Privacy and Security Assessment | ||
|
|
||
| Upon receiving a request, the [GitHub Administration Team] conducts a privacy | ||
| and security assessment of the tool. The assessment documents: | ||
|
|
||
| - What GitHub permissions and OAuth scopes the tool requires | ||
| - What data the tool accesses and where it is sent | ||
| - What AI models are used to process the code | ||
| - Data retention and deletion policies | ||
| - Security certifications (SOC2, etc.) | ||
| - Whether access can be scoped to specific repositories | ||
|
|
||
| The assessment is documented in the [kubernetes/org] issue for transparency. | ||
|
|
||
| ## Approval | ||
|
|
||
| The [GitHub Administration Team] reviews the request and the privacy and | ||
| security assessment, and approves or rejects the request. If approved, the | ||
| [GitHub Administration Team] enables the tool on the requested repositories | ||
| and applies an org-wide default configuration. | ||
|
|
||
| ## Pilot Structure | ||
|
|
||
| - The pilot runs for a 90-day evaluation period starting from the date the | ||
| tool is enabled | ||
| - The tool is enabled only on the specific requested repositories, not org-wide | ||
| - An org-wide default configuration is applied; repositories may customize | ||
| within those bounds | ||
| - The sponsoring subproject is responsible for collecting feedback from | ||
| contributors and reviewers during the pilot | ||
|
|
||
| ## Evaluation and Decision | ||
|
|
||
| At the end of the pilot period, the sponsoring subproject posts an evaluation | ||
| summary to the original [kubernetes/org] tracking issue. The summary should | ||
| cover: | ||
|
|
||
| - Quality of reviews (signal vs noise, with examples) | ||
| - Contributor and reviewer feedback | ||
| - Any issues encountered | ||
| - Recommendation (continue, modify, or remove) | ||
|
|
||
| The [GitHub Administration Team], in consultation with the sponsoring | ||
| subproject, decides to: | ||
|
|
||
| - Continue and expand to additional repositories | ||
| - Continue with modifications | ||
| - Remove the tool | ||
|
|
||
| Expansion to additional repositories requires a new issue on [kubernetes/org]. | ||
| If the tool has already been approved and assessed, the privacy and security | ||
| assessment does not need to be repeated. Requests to enable an already-approved | ||
| tool on additional repositories are evaluated on cost impact and repository | ||
| fit. Approvals for tools with per-repository costs may require additional | ||
| review. | ||
|
|
||
| ## Approved Tools | ||
|
|
||
| The following tools have completed the evaluation process and are approved for | ||
| use in the Kubernetes project. Subprojects may request enablement on their | ||
| repositories without repeating the privacy and security assessment. | ||
|
|
||
| No tools have completed the evaluation process yet. This section will be | ||
| updated as tools are approved. | ||
|
|
||
| ## Removal | ||
|
|
||
| If a pilot is unsuccessful or a tool is no longer desired, the | ||
| [GitHub Administration Team] will disable the integration. Subproject leads may | ||
| request removal at any time by filing an issue on [kubernetes/org]. | ||
|
|
||
| [GitHub Administration Team]: /github-management/README.md#github-administration-team | ||
| [AI guidance for pull requests]: /contributors/guide/pull-requests.md#ai-guidance | ||
| [OWNERS]: /contributors/guide/owners.md | ||
| [kubernetes/org]: https://github.com/kubernetes/org/issues | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what about open source tools leveraging AI ? we could have something built in the open just for issue triage for example.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The CNCF provides https://dosu.dev/ for this purpose already. ref: https://contribute.cncf.io/resources/services/hosted-tools/#tools
But imo I'm not sure this policy should be in scope for your question.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think this matters if it's open source or not -- it's more about the role in the software development lifecycle.. specifically code review.