Skip to content
Open
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 32 additions & 5 deletions cuda_bindings/build_hooks.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,13 +33,40 @@
_extensions = None


# Please keep in sync with the copy in cuda_core/build_hooks.py.
def _import_get_cuda_path_or_home():
"""Import get_cuda_path_or_home, working around PEP 517 namespace shadowing.

In isolated build environments, backend-path=["."] causes the ``cuda``
namespace package to resolve to only the project's ``cuda/`` directory,
hiding ``cuda.pathfinder`` installed in the build-env's site-packages.
Fix by replacing ``cuda.__path__`` with a plain list that includes the
site-packages ``cuda/`` directory.
"""
try:
import cuda.pathfinder
except ModuleNotFoundError:
Copy link
Copy Markdown
Contributor

@cpcloud cpcloud Mar 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This except ModuleNotFoundError is too broad: it will also catch ModuleNotFoundErrors raised from inside cuda.pathfinder itself. If cuda.pathfinder is installed but one of its own imports is broken, we silently reinterpret that as the namespace-shadowing case and take the workaround path.

I'd narrow the fallback to the exact failure we expect here:

try:
    from cuda.pathfinder import get_cuda_path_or_home
    return get_cuda_path_or_home
except ModuleNotFoundError as exc:
    if exc.name != "cuda.pathfinder":
        raise

That preserves the real traceback for genuine cuda.pathfinder import bugs and keeps the workaround limited to the cuda.pathfinder-not-found case.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for catching this (no pun intended)!

Done: commit b779d0d

import cuda

for p in sys.path:
sp_cuda = os.path.join(p, "cuda")
if os.path.isdir(os.path.join(sp_cuda, "pathfinder")):
cuda.__path__ = list(cuda.__path__) + [sp_cuda]
Copy link
Copy Markdown
Contributor

@cpcloud cpcloud Mar 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mutating cuda.__path__ here permanently changes namespace resolution for the rest of the backend process, and it selects whichever cuda/pathfinder directory happens to appear first on sys.path. That's pretty fragile.

If we keep this overall approach, I think the safer version is to resolve the installed cuda-pathfinder distribution explicitly and append only that exact cuda/ directory, instead of scanning sys.path:

from importlib import metadata
from pathlib import Path

# after narrowing the ModuleNotFoundError as above
import cuda

dist = metadata.distribution("cuda-pathfinder")
site_cuda = str(dist.locate_file(Path("cuda")))
cuda_paths = list(cuda.__path__)
if site_cuda not in cuda_paths:
    cuda.__path__ = cuda_paths + [site_cuda]

from cuda.pathfinder import get_cuda_path_or_home
return get_cuda_path_or_home

That still uses the namespace workaround, but it removes the sys.path guesswork.

If you'd rather avoid cuda.__path__ mutation entirely, another option is to load cuda/pathfinder/_utils/env_vars.py directly from the installed distribution under a private module name and call get_cuda_path_or_home() from there. That file is stdlib-only today, so it avoids rewriting namespace-package state.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the safer version is to resolve the installed cuda-pathfinder distribution explicitly

Yes, much better, thanks!

Done: commit 6facaa5

another option is to load cuda/pathfinder/_utils/env_vars.py directly

The answer to this goes deeper, I rewrote the PR description to make this clearer. Quoting from there:

The main point of this PR is to firmly establish cuda-pathfinder as a tool usable at build time, not just at runtime.

We don't want to advertise a helper function that reaches into pathfinder internals.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately it looks like we have to backtrack to the sys.path scan. Here is a Cursor writeup, based on information generated via commit 3cb0313:


We tried replacing the sys.path scan with importlib.metadata per your suggestion. It works locally but fails in CI (cibuildwheel / manylinux). Here's why.

Diagnostic output (linux-64, py3.12)

dist._path: /project/cuda_bindings/cuda_bindings.egg-info
dist._path.parent: /project/cuda_bindings
locate_file('cuda'): /project/cuda_bindings/cuda
locate_file exists: True
locate_file/pathfinder exists: False
cuda.__path__ (before): _NamespacePath(['/project/cuda_bindings/cuda'])
sys.path:
  /tmp/build-env-_bna8wjb/lib/python3.12/site-packages  ->  cuda/pathfinder exists: True
cuda.__path__ (after): _NamespacePath(['/project/cuda_bindings/cuda'])

What happens

  1. distribution("cuda-pathfinder") finds the project's own cuda_bindings.egg-info (created by the setuptools metadata step), not the cuda-pathfinder package installed in the build env under /tmp/build-env-…/lib/python3.12/site-packages/.

  2. So dist.locate_file(Path("cuda")) returns /project/cuda_bindings/cuda — the shadowed directory itself. Adding it to cuda.__path__ is a no-op.

  3. Separately, cuda.__path__ remained a _NamespacePath after our plain-list assignment. The _NamespacePath descriptor recalculates on access, discarding the mutation. (The old working code assigned cuda.__path__ = list(cuda.__path__) + [sp_cuda] without a preceding import cuda that could re-trigger recalculation.)

Conclusion

importlib.metadata is unreliable here because the in-tree egg-info shadows the installed distribution. The sys.path scan is the correct approach — it directly verifies that cuda/pathfinder/ exists on disk and is not fooled by metadata from the wrong package.

CI run with diagnostics: https://github.com/NVIDIA/cuda-python/actions/runs/23809723643

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another Cursor writeup, based on the diagnostic results:


Regarding the two concerns about the sys.path scan:

"First hit wins" ordering: In this context there is exactly one cuda-pathfinder installed in the build environment — pip deduplicates. So there is exactly one sys.path entry with cuda/pathfinder/ on disk. "First hit" finds the only hit.

Sticky cuda.__path__ mutation: The build backend is an ephemeral subprocess that builds one wheel and exits. The mutation corrects an already-wrong _NamespacePath (which only points to the project's cuda/ directory) to what it should have been (both the project's and site-packages' cuda/ directories). "Permanent" means the remaining ~seconds of that subprocess's lifetime.

And as shown in the diagnostic run, importlib.metadata cannot be used as an alternative because the in-tree cuda_bindings.egg-info shadows the installed cuda-pathfinder distribution — distribution("cuda-pathfinder") resolves to the project's own metadata, not the build-env's site-packages copy.

break
Comment on lines +52 to +56
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Path hacking of any sort like this is a code smell. At the very least the docstring is missing a concrete example of why this is needed.

isolated build environments

is doing a ton of work there. Isolated how? What is backend-path and where does that come from? Where are all these nouns like backend-path and build-env coming from?

In what real-world scenarios does this actually happen where there's not a viable alternative?

I'm really skeptical that path hacking like this is necessary.

Copy link
Copy Markdown
Collaborator Author

@rwgk rwgk Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's fully documented now here: #1824 (that in turn is pointing to #1803, where you can see the original development history for the workaround)

The comment here was updated:

commit 6892ae0

else:
raise ModuleNotFoundError(
"cuda-pathfinder is not installed in the build environment. "
"Ensure 'cuda-pathfinder>=1.5' is in build-system.requires."
Comment on lines +59 to +60
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Q: Does this msg make sense? We already updated pyproject.toml below. I suppose you meant to say: "--no-build-isolation is used, but cuda-pathfinder is not installed in the build environment." ? Because if the build isolation is in use, the build-time dependency is taken care of by the build frontend.

)
import cuda.pathfinder

return cuda.pathfinder.get_cuda_path_or_home


@functools.cache
def _get_cuda_path() -> str:
# Not using cuda.pathfinder.get_cuda_path_or_home() here because this
# build backend runs in an isolated venv where the cuda namespace package
# from backend-path shadows the installed cuda-pathfinder. See #1803 for
# a workaround to apply after cuda-pathfinder >= 1.5 is released.
cuda_path = os.environ.get("CUDA_PATH", os.environ.get("CUDA_HOME"))
get_cuda_path_or_home = _import_get_cuda_path_or_home()
cuda_path = get_cuda_path_or_home()
Comment on lines 68 to +70
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am concerned about audibility. With pathfinder being used at build time, it could be possible that if CUDA headers are installed via pip or conda, they'd be used, but we are less certain (than using CUDA_PATH/CUDA_HOME) about their version, which matters for building bindings/core. We need a way to make it auditable.

In CuPy the treatment we introduced was to hard-wire CUDA_VERSION in the extension modules:

so that we can inspect it at test/release times. Can we do the same thing?

if not cuda_path:
raise RuntimeError("Environment variable CUDA_PATH or CUDA_HOME is not set")
print("CUDA path:", cuda_path)
Expand Down
3 changes: 2 additions & 1 deletion cuda_bindings/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ requires = [
"setuptools_scm[simple]>=8",
"cython>=3.2,<3.3",
"pyclibrary>=0.1.7",
"cuda-pathfinder>=1.5",
]
build-backend = "build_hooks"
backend-path = ["."]
Expand All @@ -31,7 +32,7 @@ classifiers = [
"Environment :: GPU :: NVIDIA CUDA",
]
dynamic = ["version", "readme"]
dependencies = ["cuda-pathfinder >=1.4.2"]
dependencies = ["cuda-pathfinder >=1.5"]
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: We don't need to bump the run-time dependency; build-time (as done above) is enough


[project.optional-dependencies]
all = [
Expand Down
37 changes: 32 additions & 5 deletions cuda_core/build_hooks.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,13 +28,40 @@
COMPILE_FOR_COVERAGE = bool(int(os.environ.get("CUDA_PYTHON_COVERAGE", "0")))


# Please keep in sync with the copy in cuda_bindings/build_hooks.py.
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should avoid duplicating implementations. Could we not hoist this into a separate file and import from both?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should avoid duplicating implementations.

Agreed

Could we not hoist this into a separate file and import from both?

No 😭


In the words of Cursor Opus 4.6 1M Thinking:

There's no reasonable way to share code between the two build_hooks.py files:

  • Each is a PEP 517 build backend loaded via backend-path=["."] from its own package directory (cuda_bindings/ or cuda_core/). They can't import from the monorepo root or from each other.
  • The function exists because cuda.pathfinder can't be imported normally in this context — so it can't live inside cuda-pathfinder either.
  • A shared helper package in build-system.requires would be massive overkill for ~15 lines of stable code.

The only alternatives are worse: a new PyPI package just for this, importlib file-path hacks, or copying a shared file into both directories at CI time.

The "please keep in sync" comments are the pragmatic solution here.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have other existing cases of "please keep in sync" that are impractical to avoid.

It just crossed my mind: a pre-commit check to enforce that they stay in sync is probably pretty easy to implement.

def _import_get_cuda_path_or_home():
"""Import get_cuda_path_or_home, working around PEP 517 namespace shadowing.

In isolated build environments, backend-path=["."] causes the ``cuda``
namespace package to resolve to only the project's ``cuda/`` directory,
hiding ``cuda.pathfinder`` installed in the build-env's site-packages.
Fix by replacing ``cuda.__path__`` with a plain list that includes the
site-packages ``cuda/`` directory.
"""
try:
import cuda.pathfinder
except ModuleNotFoundError:
import cuda

for p in sys.path:
sp_cuda = os.path.join(p, "cuda")
if os.path.isdir(os.path.join(sp_cuda, "pathfinder")):
cuda.__path__ = list(cuda.__path__) + [sp_cuda]
break
else:
raise ModuleNotFoundError(
"cuda-pathfinder is not installed in the build environment. "
"Ensure 'cuda-pathfinder>=1.5' is in build-system.requires."
)
import cuda.pathfinder
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we include some message explaining that we iterated through sys.path looking for pathfinder and couldn't find it? That seems to be important information that would be missing from just throwing we couldn't import pathfinder module.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

          for p in sys.path:
              sp_cuda = os.path.join(p, "cuda")                                                                                
              if os.path.isdir(os.path.join(sp_cuda, "pathfinder")):
                  cuda.__path__ = list(cuda.__path__) + [sp_cuda]                                                              
                  break                 
          else:                                                                                                                
              raise ModuleNotFoundError(
                  "cuda-pathfinder is not installed in the build environment. "                                                
                  "Ensure 'cuda-pathfinder>=1.5' is in build-system.requires."                                                 
              )
          import cuda.pathfinder ```

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done: commit 3d422de


return cuda.pathfinder.get_cuda_path_or_home


@functools.cache
def _get_cuda_path() -> str:
# Not using cuda.pathfinder.get_cuda_path_or_home() here because this
# build backend runs in an isolated venv where the cuda namespace package
# from backend-path shadows the installed cuda-pathfinder. See #1803 for
# a workaround to apply after cuda-pathfinder >= 1.5 is released.
cuda_path = os.environ.get("CUDA_PATH", os.environ.get("CUDA_HOME"))
get_cuda_path_or_home = _import_get_cuda_path_or_home()
cuda_path = get_cuda_path_or_home()
if not cuda_path:
raise RuntimeError("Environment variable CUDA_PATH or CUDA_HOME is not set")
print("CUDA path:", cuda_path)
Expand Down
3 changes: 2 additions & 1 deletion cuda_core/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ requires = [
"setuptools>=80",
"setuptools-scm[simple]>=8",
"Cython>=3.2,<3.3",
"cuda-pathfinder>=1.5"
]
build-backend = "build_hooks"
backend-path = ["."]
Expand Down Expand Up @@ -47,7 +48,7 @@ classifiers = [
"Environment :: GPU :: NVIDIA CUDA :: 13",
]
dependencies = [
"cuda-pathfinder >=1.4.2",
"cuda-pathfinder >=1.5",
"numpy",
]

Expand Down
7 changes: 6 additions & 1 deletion cuda_core/tests/test_build_hooks.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# SPDX-FileCopyrightText: Copyright (c) 2024-2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-FileCopyrightText: Copyright (c) 2024-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

"""Tests for build_hooks.py build infrastructure.
Expand All @@ -24,6 +24,8 @@

import pytest

from cuda.pathfinder import get_cuda_path_or_home

# build_hooks.py imports Cython and setuptools at the top level, so skip if not available
pytest.importorskip("Cython")
pytest.importorskip("setuptools")
Expand Down Expand Up @@ -68,6 +70,7 @@ def _check_version_detection(

build_hooks._get_cuda_path.cache_clear()
build_hooks._determine_cuda_major_version.cache_clear()
get_cuda_path_or_home.cache_clear()

mock_env = {
k: v
Expand All @@ -92,6 +95,7 @@ def test_env_var_override(self, version):
"""CUDA_CORE_BUILD_MAJOR env var override works with various versions."""
build_hooks._get_cuda_path.cache_clear()
build_hooks._determine_cuda_major_version.cache_clear()
get_cuda_path_or_home.cache_clear()
with mock.patch.dict(os.environ, {"CUDA_CORE_BUILD_MAJOR": version}, clear=False):
result = build_hooks._determine_cuda_major_version()
assert result == version
Expand Down Expand Up @@ -125,6 +129,7 @@ def test_missing_cuda_path_raises_error(self):
"""RuntimeError is raised when CUDA_PATH/CUDA_HOME not set and no env var override."""
build_hooks._get_cuda_path.cache_clear()
build_hooks._determine_cuda_major_version.cache_clear()
get_cuda_path_or_home.cache_clear()
with (
mock.patch.dict(os.environ, {}, clear=True),
pytest.raises(RuntimeError, match="CUDA_PATH or CUDA_HOME"),
Expand Down
2 changes: 1 addition & 1 deletion cuda_pathfinder/pixi.toml
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ cu12 = { features = ["cu12", "test"], solve-group = "cu12" }
# TODO: check if these can be extracted from pyproject.toml
[package]
name = "cuda-pathfinder"
version = "1.3.4a0"
version = "1.5.0"

[package.build]
backend = { name = "pixi-build-python", version = "*" }
Expand Down
Loading