Skip to content

[MAHOUT-1414][QDP] Expose native cuda_available() and gate GPU tests on it#1416

Merged
400Ping merged 1 commit into
mainfrom
qdp-unify-gpu-detection
Jul 5, 2026
Merged

[MAHOUT-1414][QDP] Expose native cuda_available() and gate GPU tests on it#1416
400Ping merged 1 commit into
mainfrom
qdp-unify-gpu-detection

Conversation

@ryankert01

@ryankert01 ryankert01 commented Jun 26, 2026

Copy link
Copy Markdown
Member

Closes #1414.

Why

#1321 lets _qdp build and import without the CUDA toolkit (stub CUDA Runtime symbols). That breaks the long-standing assumption that "the _qdp extension imports" ⟺ "a GPU is usable", which the test suite relied on to gate GPU tests. #1321 patched the immediate breakage by skipping @pytest.mark.gpu tests on torch.cuda.is_available(), but that is a proxy: on this feature's headline scenario — a GPU host with PyTorch but no toolkit — _qdp is a stub build, yet torch.cuda.is_available() is still True, so the GPU tests would run against the stub and abort.

This PR makes the engine report the truth.

What

  • qdp-core: add cuda_runtime_available() — queries cudaGetDeviceCount and returns true only when a device is actually present. In a qdp_no_cuda (stub) build it resolves to the existing 999-sentinel stub, so it returns false with no cfg branching or dead code. Re-exported from the crate root.
  • _qdp: expose it as _qdp.cuda_available().
  • qumat_qdp: add is_cuda_available() (mirrors the existing is_triton_amd_available()) as the single Python source of truth.
  • testing/conftest.py: gate the @pytest.mark.gpu auto-skip on the native signal, falling back to torch only if the helper is absent.
  • test_fallback.py: coverage that also runs on a stub build, guarding that querying availability returns a bool without aborting.

Test plan

Verified locally on 4× NVIDIA GPUs (CUDA 12.8), real build:

  • _qdp.cuda_available() / qumat_qdp.is_cuda_available()True with a GPU, False under CUDA_VISIBLE_DEVICES="".
  • testing/qdp + testing/qdp_python with CUDA hidden: 130 passed, 156 skipped, 0 failed (no aborts).
  • The 10 tests that crashed in fix(qdp-core): link cudart locally and stub FFI when toolkit is absent  #1321's CI run: pass on GPU, skip when hidden.
  • qdp-core builds in both real and QDP_NO_CUDA=1 stub modes; full Rust suite, clippy --all-features -D warnings, ruff, and ty all clean.

Out of scope (left in #1414)

Making the stub path raise a clean RuntimeError instead of aborting for direct (non-test) callers, and collapsing the ~65 redundant inline torch.cuda.is_available() checks now that the conftest gate is accurate.

@ryankert01
ryankert01 requested a review from 400Ping as a code owner June 26, 2026 19:13
@ryankert01
ryankert01 requested a review from rich7420 June 26, 2026 19:26
Follow-up to #1321 (addresses #1414). After #1321, `_qdp` can build and
import without the CUDA toolkit (stub runtime), so "extension importable"
no longer implies a usable GPU. The test suite gated GPU tests on
`torch.cuda.is_available()`, a proxy that is wrong on this PR's headline
scenario -- a GPU host with PyTorch but no toolkit, where `_qdp` is a stub
build yet torch still reports a device.

- qdp-core: add `cuda_runtime_available()`, which queries
  `cudaGetDeviceCount` (false in a stub build via the existing 999 sentinel
  stub, and on hosts with no device). Re-exported from the crate root.
- _qdp: expose it as `_qdp.cuda_available()`.
- qumat_qdp: add `is_cuda_available()`, mirroring `is_triton_amd_available()`,
  as the single Python source of truth.
- testing/conftest: gate the `@pytest.mark.gpu` auto-skip on the native
  signal (falling back to torch only if the helper is absent).
- test_fallback: coverage that runs on a stub build too, guarding that
  querying availability returns a bool without aborting.

Verified on GPU (tests run and pass) and with CUDA hidden (tests skip);
full Rust suite, clippy --all-features, ruff, and ty all clean.
@ryankert01
ryankert01 force-pushed the qdp-unify-gpu-detection branch from 1c9078b to bffb3ed Compare June 28, 2026 15:50
@rich7420

rich7420 commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

@ryankert01 thanks for the patch, Nice one
locally tested
the no-cfg-branch stub resolve is clean, and gating on the native signal instead of torch is the right fix. Verified both crates build in stub mode.

Two tiny nits, both optional:

  1. Three names for one concept (cuda_runtime_availablecuda_availableis_cuda_available). Each follows local convention so it's fine, just a little to track.(it's not that important)

  2. is_cuda_available() does a fresh FFI cudaGetDeviceCount every call, and conftest calls it per gpu test. Cheap so not a perf issue, but the result can't change within a run — an lru_cache (or OnceLock<bool>) would make it a one-time query.

@viiccwen viiccwen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@400Ping 400Ping changed the title [QDP] Expose native cuda_available() and gate GPU tests on it [MAHOUT-1414][QDP] Expose native cuda_available() and gate GPU tests on it Jul 5, 2026
@400Ping
400Ping merged commit 826c3ab into main Jul 5, 2026
8 checks passed
@400Ping
400Ping deleted the qdp-unify-gpu-detection branch July 5, 2026 12:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[QDP] Unify GPU-availability detection in tests; stub CUDA path should raise, not abort

4 participants