Fix cache warmer: run real test to resolve surefire provider#1385
Merged
gopalldb merged 25 commits intodatabricks:mainfrom Apr 8, 2026
Merged
Fix cache warmer: run real test to resolve surefire provider#1385gopalldb merged 25 commits intodatabricks:mainfrom
gopalldb merged 25 commits intodatabricks:mainfrom
Conversation
Reverts databricks#1350. GitHub-hosted runner IPs are blocked by the Databricks org IP allow list, causing gh CLI API calls to fail and preventing the required status checks from matching (runner label mismatch). Restores all 22 workflow files to use databricks-protected-runner-group with linux-ubuntu-latest and windows-server-latest labels. Also configures JFrog Artifactory as Maven mirror via OIDC token exchange, since Databricks runners cannot access public registries directly (supply chain security policy). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Gopal Lal <gopal.lal@databricks.com>
Forked PRs cannot authenticate to JFrog Artifactory via OIDC (GitHub
restricts id-token for fork workflows). This change enables forked PR
CI by pre-caching dependencies from a privileged workflow.
New files:
- .github/actions/setup-maven/action.yml: Reusable composite action
that detects forked PRs and either authenticates to JFrog (same-repo)
or restores the dependency cache (fork)
- .github/workflows/warmMavenCache.yml: Privileged workflow that
resolves all dependencies via JFrog and saves the cache. Triggers on
pom.xml changes to main, daily schedule, and manual dispatch with
optional PR number for warming from a fork's pom.xml
Modified workflows to use the composite action:
- prCheck.yml (formatting, unit tests, packaging)
- prIntegrationTests.yml
- coverageReport.yml
Cache key: {os}-maven-deps-{hash(pom.xml)} with prefix restore-keys.
Forked PRs read cache from the default branch per GitHub Actions rules.
Signed-off-by: Gopal Lal <gopal.lal@databricks.com>
Co-authored-by: Isaac
Signed-off-by: Gopal Lal <gopal.lal@databricks.com>
When a PR with dependency changes merges to main, the cache warmer now runs a cleanup job that deletes maven-deps cache entries from previous pom.xml versions. This prevents stale dependency caches from occupying space when concurrent PRs have different dependency versions. Cache lifecycle: - Each unique pom.xml hash gets its own cache entry (content-addressable) - Multiple concurrent PRs coexist in cache with different keys - On merge to main, stale entries (not matching main's current hash) are deleted via gh cache delete - GitHub also auto-evicts caches not accessed in 7 days Signed-off-by: Gopal Lal <gopal.lal@databricks.com> Co-authored-by: Isaac Signed-off-by: Gopal Lal <gopal.lal@databricks.com>
GitHub Actions automatically evicts cache entries not accessed in 7 days. This is sufficient for cleaning up stale PR dependency caches. The explicit cleanup job adds complexity without meaningful benefit. Signed-off-by: Gopal Lal <gopal.lal@databricks.com> Co-authored-by: Isaac Signed-off-by: Gopal Lal <gopal.lal@databricks.com>
…utionTests - prCheckJDK8.yml: Add fork detection + composite action (triggers on pull_request to jdk-8 branch, so forked PRs are affected) - concurrencyExecutionTests.yml: Use composite action for consistency (always is-fork=false since it only triggers on push/dispatch) Both workflows now use .github/actions/setup-maven instead of inline JFrog OIDC + cache boilerplate. Signed-off-by: Gopal Lal <gopal.lal@databricks.com> Co-authored-by: Isaac Signed-off-by: Gopal Lal <gopal.lal@databricks.com>
- Drop Windows from cache warmer matrix — Windows runners in
databricks-protected-runner-group lack bash (command not found)
- Remove runner.os from cache key — Maven JARs/POMs are platform-
independent, so one cache entry serves both Linux and Windows
- Cache key is now: maven-deps-{hash(pom.xml)}
Signed-off-by: Gopal Lal <gopal.lal@databricks.com>
Co-authored-by: Isaac
Signed-off-by: Gopal Lal <gopal.lal@databricks.com>
- Drop Windows from cache warmer matrix — Windows runners in
databricks-protected-runner-group lack bash (command not found)
- Remove runner.os from cache key — Maven JARs/POMs are platform-
independent, so one cache entry serves both Linux and Windows
- Cache key is now: maven-deps-{hash(pom.xml)}
Signed-off-by: Gopal Lal <gopal.lal@databricks.com>
Co-authored-by: Isaac
Signed-off-by: Gopal Lal <gopal.lal@databricks.com>
Keep -Ddependency-check.skip=true from main in the coverage test command. Signed-off-by: Gopal Lal <gopal.lal@databricks.com> Co-authored-by: Isaac Signed-off-by: Gopal Lal <gopal.lal@databricks.com>
The thin/uber jar modules depend on databricks-jdbc-core SNAPSHOT which must be installed into ~/.m2/repository first. Changed from mvn compile to mvn install -DskipTests so inter-module SNAPSHOT artifacts are available during dependency resolution. Signed-off-by: Gopal Lal <gopal.lal@databricks.com> Co-authored-by: Isaac Signed-off-by: Gopal Lal <gopal.lal@databricks.com>
dependency:resolve fails on inter-module SNAPSHOTs (databricks-jdbc-core) because they don't exist in JFrog — they're local build artifacts. Since dependency:resolve runs first with set -euo pipefail, the install command never executes. Fix: use mvn install alone, which handles both external dependency resolution from JFrog AND inter-module SNAPSHOT installation. Signed-off-by: Gopal Lal <gopal.lal@databricks.com> Co-authored-by: Isaac Signed-off-by: Gopal Lal <gopal.lal@databricks.com>
Keep install-only approach (no dependency:resolve) to avoid inter-module SNAPSHOT resolution failures. Signed-off-by: Gopal Lal <gopal.lal@databricks.com> Co-authored-by: Isaac Signed-off-by: Gopal Lal <gopal.lal@databricks.com>
The cache was missing maven-toolchains-plugin (and potentially other plugins like spotless, jacoco) because mvn install only resolves plugins needed for the install lifecycle. Plugins activated by specific goals or profiles (used in PR unit-test and formatting workflows) were not cached, causing 401 errors for forked PRs. Fix: after install, also run dependency:resolve-plugins and trigger spotless/jacoco plugin downloads to ensure all PR workflow dependencies are cached. Signed-off-by: Gopal Lal <gopal.lal@databricks.com> Co-authored-by: Isaac Signed-off-by: Gopal Lal <gopal.lal@databricks.com>
The cache was missing test-time artifacts (surefire-junit-platform, jacoco agent, spotless, toolchains plugin, maven-metadata.xml) because mvn install -DskipTests only resolves compile-time dependencies. Fix: after install, run the same Maven commands that PR workflows use (with a no-op test filter) to trigger resolution of all plugins and providers. This covers: - surefire-junit-platform (resolved at test execution time) - maven-toolchains-plugin (resolved when toolchains goal is active) - spotless plugin + formatters (resolved during spotless:check) - jacoco agent + report plugins (resolved during jacoco:report) - plugin group maven-metadata.xml files Signed-off-by: Gopal Lal <gopal.lal@databricks.com> Co-authored-by: Isaac Signed-off-by: Gopal Lal <gopal.lal@databricks.com>
Windows runners in databricks-protected-runner-group don't have git pre-installed. The "Enable long paths" step ran before actions/checkout (which installs git), so "git config --system core.longpaths true" failed with "git: command not found". Fix: use Windows registry (New-ItemProperty LongPathsEnabled) which doesn't require git. Also attempt git config as fallback if git is available. Signed-off-by: Gopal Lal <gopal.lal@databricks.com> Co-authored-by: Isaac Signed-off-by: Gopal Lal <gopal.lal@databricks.com>
Databricks protected Windows runners (windows-server-latest) don't have git pre-installed, causing failures at: 1. "Enable long paths" (git config --system) 2. actions/checkout (requires git) 3. shell: bash steps (requires git bash) Fix: download and install PortableGit from git-for-windows before any git-dependent steps. This provides git.exe, bash.exe, and standard Unix utilities. Also enables long paths via both git config and Windows registry. Signed-off-by: Gopal Lal <gopal.lal@databricks.com> Co-authored-by: Isaac Signed-off-by: Gopal Lal <gopal.lal@databricks.com>
Windows runners in databricks-protected-runner-group lack git. This is a pre-existing issue to be resolved with the runner team. Keep Windows in the matrix so failures are visible. Signed-off-by: Gopal Lal <gopal.lal@databricks.com> Co-authored-by: Isaac Signed-off-by: Gopal Lal <gopal.lal@databricks.com>
Keep multi-step dependency resolution approach that caches all plugins (surefire, spotless, jacoco, toolchains). Signed-off-by: Gopal Lal <gopal.lal@databricks.com> Co-authored-by: Isaac Signed-off-by: Gopal Lal <gopal.lal@databricks.com>
…tion surefire-junit-platform is resolved lazily at test execution time, not at plugin initialization. The previous approach (-Dtest=NoSuchTest) failed before surefire downloaded the provider JAR, so it was never cached. Fix: run a real lightweight test (DatabricksParameterMetaDataTest# testInitialization) to force surefire to fully resolve and download its JUnit platform provider. Signed-off-by: Gopal Lal <gopal.lal@databricks.com> Co-authored-by: Isaac Signed-off-by: Gopal Lal <gopal.lal@databricks.com>
tejassp-db
approved these changes
Apr 8, 2026
Signed-off-by: Gopal Lal <gopal.lal@databricks.com> Co-authored-by: Isaac Signed-off-by: Gopal Lal <gopal.lal@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The cache warmer was not caching
surefire-junit-platform:3.1.2because surefire resolves its provider JAR lazily at test execution time, not at plugin initialization. The previous approach (-Dtest=NoSuchTest) failed before surefire downloaded the provider.Fix: run a real lightweight test (
DatabricksParameterMetaDataTest#testInitialization) to force full surefire provider resolution.Verified locally
All 4 previously missing artifacts are now in the local cache after running the test:
surefire-junit-platform:3.1.2(blocked unit-tests + integration tests)maven-toolchains-plugin:3.2.0(blocked unit-tests)maven-clean-plugin:3.2.0(blocked coverage)exec-maven-plugin:1.2.1(blocked coverage)The 5 maven-metadata.xml warnings are non-blocking (Maven falls back to cached plugin JARs).
Test plan
mvn -pl jdbc-core test -Dtest="DatabricksParameterMetaDataTest#testInitialization"passes and resolves all missing pluginsNO_CHANGELOG=true
This pull request was AI-assisted by Isaac.