[CI] Add Initial Wiring for Premerge Advisor Explanations

[boomanaiden154]

This patch adds a new script, premerge_advisor_explain.py that requests test failure explanations from the premerge advisor. For now it just prints them out to STDOUT. This allows for testing of the entire system by looking at failure explanations in failed jobs before we do the rest of the wiring to enable the premerge advisor to write out comments.

[llvmbot]

@llvm/pr-subscribers-tablegen

@llvm/pr-subscribers-infrastructure

Author: Aiden Grossman (boomanaiden154)

Changes

This patch adds a new script, premerge_advisor_explain.py that requests test failure explanations from the premerge advisor. For now it just prints them out to STDOUT. This allows for testing of the entire system by looking at failure explanations in failed jobs before we do the rest of the wiring to enable the premerge advisor to write out comments.

---

Full diff: https://github.com/llvm/llvm-project/pull/164132.diff

2 Files Affected:

• (added) .ci/premerge_advisor_explain.py (+60)
• (modified) .ci/utils.sh (+5)

diff --git a/.ci/premerge_advisor_explain.py b/.ci/premerge_advisor_explain.py
new file mode 100644
index 0000000000000..d6493635976b1
--- /dev/null
+++ b/.ci/premerge_advisor_explain.py
@@ -0,0 +1,60 @@
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+"""Script for getting explanations from the premerge advisor."""
+
+import argparse
+import os
+import platform
+import sys
+
+import requests
+
+import generate_test_report_lib
+
+PREMERGE_ADVISOR_URL = (
+    "http://premerge-advisor.premerge-advisor.svc.cluster.local:5000/explain"
+)
+
+
+def main(commit_sha: str, build_log_files: list[str]):
+    junit_objects, ninja_logs = generate_test_report_lib.load_info_from_files(
+        build_log_files
+    )
+    test_failures = generate_test_report_lib.get_failures(junit_objects)
+    current_platform = f"{platform.system()}-{platform.machine()}".lower()
+    explanation_reuqest = {
+        "base_commit_sha": commit_sha,
+        "platform": current_platform,
+        "failures": [],
+    }
+    if test_failures:
+        for _, failures in test_failures.items():
+            for name, failure_messsage in failures:
+                explanation_reuqest["failures"].append(
+                    {"name": name, "message": failure_messsage}
+                )
+    else:
+        ninja_failures = generate_test_report_lib.find_failure_in_ninja_logs(ninja_logs)
+        for name, failure_message in ninja_failures:
+            explanation_reuqest["failures"].append(
+                {"name": name, "message": failure_message}
+            )
+    advisor_response = requests.get(PREMERGE_ADVISOR_URL, json=explanation_reuqest)
+    print(advisor_response.json)
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument("commit_sha", help="The base commit SHA for the test.")
+    parser.add_argument(
+        "build_log_files", help="Paths to JUnit report files and ninja logs.", nargs="*"
+    )
+    args = parser.parse_args()
+
+    # Skip looking for results on AArch64 for now because the premerge advisor
+    # service is not available on AWS currently.
+    if platform.machine() == "arm64":
+        sys.exit(0)
+
+    main(args.commit_sha, args.build_log_files)
diff --git a/.ci/utils.sh b/.ci/utils.sh
index 375ed2973bc53..dc8ce9b9a4214 100644
--- a/.ci/utils.sh
+++ b/.ci/utils.sh
@@ -43,6 +43,11 @@ function at-exit {
     python "${MONOREPO_ROOT}"/.ci/premerge_advisor_upload.py \
       $(git rev-parse HEAD~1) $GITHUB_RUN_NUMBER \
       "${BUILD_DIR}"/test-results.*.xml "${MONOREPO_ROOT}"/ninja*.log
+    if [[ "$GITHUB_ACTIONS" != "" ]]; then
+      python "${MONOREPO_ROOT}"/.ci/premerge_advisor_explain.py \
+        $(git rev-parse HEAD~1) "${BUILD_DIR}"/test-results.*.xml \
+        "${MONOREPO_ROOT}"/ninja*.log
+    fi
   fi
 }
 trap at-exit EXIT

[cor3ntin]

What is the pre-merge advisor?

[boomanaiden154]

What is the pre-merge advisor?

There's a mostly finished but still in progress design doc at llvm/llvm-zorg#599. Essentially it's a system that looks at premerge failures and will let you know if they're safe to ignore because they are either failing or flaky at head.