Fix fork Maven config: set updatePolicy=never to prevent 401 on cached artifacts#1389
Merged
gopalldb merged 28 commits intodatabricks:mainfrom Apr 8, 2026
Merged
Conversation
Reverts databricks#1350. GitHub-hosted runner IPs are blocked by the Databricks org IP allow list, causing gh CLI API calls to fail and preventing the required status checks from matching (runner label mismatch). Restores all 22 workflow files to use databricks-protected-runner-group with linux-ubuntu-latest and windows-server-latest labels. Also configures JFrog Artifactory as Maven mirror via OIDC token exchange, since Databricks runners cannot access public registries directly (supply chain security policy). Co-Authored-By: Claude Opus 4.6 <[email protected]> Signed-off-by: Gopal Lal <[email protected]>
Forked PRs cannot authenticate to JFrog Artifactory via OIDC (GitHub
restricts id-token for fork workflows). This change enables forked PR
CI by pre-caching dependencies from a privileged workflow.
New files:
- .github/actions/setup-maven/action.yml: Reusable composite action
that detects forked PRs and either authenticates to JFrog (same-repo)
or restores the dependency cache (fork)
- .github/workflows/warmMavenCache.yml: Privileged workflow that
resolves all dependencies via JFrog and saves the cache. Triggers on
pom.xml changes to main, daily schedule, and manual dispatch with
optional PR number for warming from a fork's pom.xml
Modified workflows to use the composite action:
- prCheck.yml (formatting, unit tests, packaging)
- prIntegrationTests.yml
- coverageReport.yml
Cache key: {os}-maven-deps-{hash(pom.xml)} with prefix restore-keys.
Forked PRs read cache from the default branch per GitHub Actions rules.
Signed-off-by: Gopal Lal <[email protected]>
Co-authored-by: Isaac
Signed-off-by: Gopal Lal <[email protected]>
When a PR with dependency changes merges to main, the cache warmer now runs a cleanup job that deletes maven-deps cache entries from previous pom.xml versions. This prevents stale dependency caches from occupying space when concurrent PRs have different dependency versions. Cache lifecycle: - Each unique pom.xml hash gets its own cache entry (content-addressable) - Multiple concurrent PRs coexist in cache with different keys - On merge to main, stale entries (not matching main's current hash) are deleted via gh cache delete - GitHub also auto-evicts caches not accessed in 7 days Signed-off-by: Gopal Lal <[email protected]> Co-authored-by: Isaac Signed-off-by: Gopal Lal <[email protected]>
GitHub Actions automatically evicts cache entries not accessed in 7 days. This is sufficient for cleaning up stale PR dependency caches. The explicit cleanup job adds complexity without meaningful benefit. Signed-off-by: Gopal Lal <[email protected]> Co-authored-by: Isaac Signed-off-by: Gopal Lal <[email protected]>
…utionTests - prCheckJDK8.yml: Add fork detection + composite action (triggers on pull_request to jdk-8 branch, so forked PRs are affected) - concurrencyExecutionTests.yml: Use composite action for consistency (always is-fork=false since it only triggers on push/dispatch) Both workflows now use .github/actions/setup-maven instead of inline JFrog OIDC + cache boilerplate. Signed-off-by: Gopal Lal <[email protected]> Co-authored-by: Isaac Signed-off-by: Gopal Lal <[email protected]>
- Drop Windows from cache warmer matrix — Windows runners in
databricks-protected-runner-group lack bash (command not found)
- Remove runner.os from cache key — Maven JARs/POMs are platform-
independent, so one cache entry serves both Linux and Windows
- Cache key is now: maven-deps-{hash(pom.xml)}
Signed-off-by: Gopal Lal <[email protected]>
Co-authored-by: Isaac
Signed-off-by: Gopal Lal <[email protected]>
- Drop Windows from cache warmer matrix — Windows runners in
databricks-protected-runner-group lack bash (command not found)
- Remove runner.os from cache key — Maven JARs/POMs are platform-
independent, so one cache entry serves both Linux and Windows
- Cache key is now: maven-deps-{hash(pom.xml)}
Signed-off-by: Gopal Lal <[email protected]>
Co-authored-by: Isaac
Signed-off-by: Gopal Lal <[email protected]>
Keep -Ddependency-check.skip=true from main in the coverage test command. Signed-off-by: Gopal Lal <[email protected]> Co-authored-by: Isaac Signed-off-by: Gopal Lal <[email protected]>
The thin/uber jar modules depend on databricks-jdbc-core SNAPSHOT which must be installed into ~/.m2/repository first. Changed from mvn compile to mvn install -DskipTests so inter-module SNAPSHOT artifacts are available during dependency resolution. Signed-off-by: Gopal Lal <[email protected]> Co-authored-by: Isaac Signed-off-by: Gopal Lal <[email protected]>
dependency:resolve fails on inter-module SNAPSHOTs (databricks-jdbc-core) because they don't exist in JFrog — they're local build artifacts. Since dependency:resolve runs first with set -euo pipefail, the install command never executes. Fix: use mvn install alone, which handles both external dependency resolution from JFrog AND inter-module SNAPSHOT installation. Signed-off-by: Gopal Lal <[email protected]> Co-authored-by: Isaac Signed-off-by: Gopal Lal <[email protected]>
Keep install-only approach (no dependency:resolve) to avoid inter-module SNAPSHOT resolution failures. Signed-off-by: Gopal Lal <[email protected]> Co-authored-by: Isaac Signed-off-by: Gopal Lal <[email protected]>
The cache was missing maven-toolchains-plugin (and potentially other plugins like spotless, jacoco) because mvn install only resolves plugins needed for the install lifecycle. Plugins activated by specific goals or profiles (used in PR unit-test and formatting workflows) were not cached, causing 401 errors for forked PRs. Fix: after install, also run dependency:resolve-plugins and trigger spotless/jacoco plugin downloads to ensure all PR workflow dependencies are cached. Signed-off-by: Gopal Lal <[email protected]> Co-authored-by: Isaac Signed-off-by: Gopal Lal <[email protected]>
The cache was missing test-time artifacts (surefire-junit-platform, jacoco agent, spotless, toolchains plugin, maven-metadata.xml) because mvn install -DskipTests only resolves compile-time dependencies. Fix: after install, run the same Maven commands that PR workflows use (with a no-op test filter) to trigger resolution of all plugins and providers. This covers: - surefire-junit-platform (resolved at test execution time) - maven-toolchains-plugin (resolved when toolchains goal is active) - spotless plugin + formatters (resolved during spotless:check) - jacoco agent + report plugins (resolved during jacoco:report) - plugin group maven-metadata.xml files Signed-off-by: Gopal Lal <[email protected]> Co-authored-by: Isaac Signed-off-by: Gopal Lal <[email protected]>
Windows runners in databricks-protected-runner-group don't have git pre-installed. The "Enable long paths" step ran before actions/checkout (which installs git), so "git config --system core.longpaths true" failed with "git: command not found". Fix: use Windows registry (New-ItemProperty LongPathsEnabled) which doesn't require git. Also attempt git config as fallback if git is available. Signed-off-by: Gopal Lal <[email protected]> Co-authored-by: Isaac Signed-off-by: Gopal Lal <[email protected]>
Databricks protected Windows runners (windows-server-latest) don't have git pre-installed, causing failures at: 1. "Enable long paths" (git config --system) 2. actions/checkout (requires git) 3. shell: bash steps (requires git bash) Fix: download and install PortableGit from git-for-windows before any git-dependent steps. This provides git.exe, bash.exe, and standard Unix utilities. Also enables long paths via both git config and Windows registry. Signed-off-by: Gopal Lal <[email protected]> Co-authored-by: Isaac Signed-off-by: Gopal Lal <[email protected]>
Windows runners in databricks-protected-runner-group lack git. This is a pre-existing issue to be resolved with the runner team. Keep Windows in the matrix so failures are visible. Signed-off-by: Gopal Lal <[email protected]> Co-authored-by: Isaac Signed-off-by: Gopal Lal <[email protected]>
Keep multi-step dependency resolution approach that caches all plugins (surefire, spotless, jacoco, toolchains). Signed-off-by: Gopal Lal <[email protected]> Co-authored-by: Isaac Signed-off-by: Gopal Lal <[email protected]>
…tion surefire-junit-platform is resolved lazily at test execution time, not at plugin initialization. The previous approach (-Dtest=NoSuchTest) failed before surefire downloaded the provider JAR, so it was never cached. Fix: run a real lightweight test (DatabricksParameterMetaDataTest# testInitialization) to force surefire to fully resolve and download its JUnit platform provider. Signed-off-by: Gopal Lal <[email protected]> Co-authored-by: Isaac Signed-off-by: Gopal Lal <[email protected]>
Signed-off-by: Gopal Lal <[email protected]> Co-authored-by: Isaac Signed-off-by: Gopal Lal <[email protected]>
Even with artifacts in the local cache, Maven checks the remote repository for updates (plugin POM metadata). For forked PRs the mirror has no credentials, so these checks get 401 errors. Fix: configure repository and pluginRepository with updatePolicy=never in an active profile for forked PRs only. This tells Maven to use cached artifacts without contacting the remote for updates. The non-fork path (JFrog OIDC) is unchanged. Signed-off-by: Gopal Lal <[email protected]> Co-authored-by: Isaac Signed-off-by: Gopal Lal <[email protected]>
vikrantpuppala
approved these changes
Apr 8, 2026
samikshya-db
approved these changes
Apr 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Even with artifacts in the local cache (~350MB restored successfully), Maven checks the remote repository for POM metadata updates. For forked PRs the JFrog mirror has no credentials, so these update checks get 401 errors — causing
maven-toolchains-plugin:3.2.0resolution to fail even though the JAR is cached.Fix: for forked PRs only, configure repository and pluginRepository with
updatePolicy=neverin an active profile. This tells Maven to use cached artifacts without contacting the remote for updates. The non-fork path (JFrog OIDC with full credentials) is unchanged.Test plan
NO_CHANGELOG=true
This pull request was AI-assisted by Isaac.