Repository navigation
[BUG]: Constructing PinnedMemoryResource() invalidates a global-mode graph capture on another thread #3451
Description
Is this a duplicate?
- I confirmed there appear to be no duplicate issues for this bug and that I agree to the Code of Conduct
Type of Bug
Runtime Error
Component
cuda.core
Describe the bug
Since #3405, constructing PinnedMemoryResource() calls cuMemPoolSetAccess to grant the current device access to the pool. The CUDA driver refuses that call while any thread is capturing a CUDA graph in global capture mode. Construction raises CUDA_ERROR_STREAM_CAPTURE_UNSUPPORTED, and the driver invalidates the other thread's capture, so end_building() on the capturing thread raises CUDA_ERROR_STREAM_CAPTURE_INVALIDATED. This happens on every construction, including when the pool already has the requested access.
At c394b5e1609, the commit just before #3405, the same steps construct the resource and the capture stays valid.
cuda.core's begin_building() defaults to relaxed mode, which is not affected. Global mode is the default in other libraries, for example PyTorch's torch.cuda.graph (capture_error_mode='global').
Construction with options, for example PinnedMemoryResourceOptions(max_size=...), fails the same way both before and after #3405, because the driver also refuses cuMemPoolCreate during a global-mode capture.
Related: #3415 proposes widening the same grant to all devices.
How to Reproduce
import threading
from cuda.core import Device, PinnedMemoryResource
dev = Device(0)
dev.set_current()
gb = dev.create_graph_builder().begin_building(mode="global")
def construct():
Device(0).set_current()
try:
PinnedMemoryResource()
print("construct: ok")
except Exception as exc:
print("construct:", exc)
t = threading.Thread(target=construct)
t.start()
t.join()
gb.end_building()
print("end_building: ok")
gb.close()Output on main at b975a83:
construct: CUDA_ERROR_STREAM_CAPTURE_UNSUPPORTED: This error indicates that the operation is not permitted when the stream is capturing.
Traceback (most recent call last):
...
File "cuda/core/graph/_graph_builder.pyx", line 554, in cuda.core.graph._graph_builder.GraphBuilder.end_building
...
cuda.core._utils.cuda_utils.CUDAError: CUDA_ERROR_STREAM_CAPTURE_INVALIDATED: This error indicates that the current capture sequence on the stream has been invalidated due to a previous error.
At c394b5e1609 (before #3405), construction succeeds and end_building() succeeds.
With begin_building(mode="relaxed"), both steps succeed on main.
Expected behavior
Constructing PinnedMemoryResource() without options does not affect a graph capture running on another thread, as before #3405.
Operating System
Ubuntu 24.04.4 LTS, Linux 7.0.0-38-generic, x86_64
nvidia-smi output
NVIDIA-SMI 595.99.02 Driver Version: 595.99.02 CUDA Version: 13.2
GPU 0: NVIDIA RTX 6000 Ada Generation, 49140 MiB, Addressing Mode: HMM
Python environment (main at b975a83, editable builds):
- Python 3.12.3
- cuda.core 1.3.1.dev10+gb975a83a9
- cuda.bindings 13.5.1.dev36+gb975a83a9
- cuda.pathfinder 1.9.1.dev2+gb975a83a9
The baseline run at c394b5e1609 used cuda.core 1.2.1.dev357+gc394b5e16.
Activity
juenglin commentedon Nov 13, 2026
Suggested fix: run the grant in relaxed capture mode, the way cuda.core already handles pool attribute reads and writes and
trim()(see_enter_relaxed_capture_modein_memory_pool.pyx). The calling thread switches itself to relaxed mode aroundcuMemPoolSetAccessand then restores its previous mode. The grant still takes effect immediately, and a global-mode capture on another thread stays valid. I checked this at the driver level (Linux, driver 595.99.02): in relaxed mode,cuMemPoolSetAccessfrom another thread succeeds and the capture ends normally.This keeps the behavior from #3405 unchanged otherwise. Allocating and freeing from the pool on another thread remain refused during a global-mode capture, as they are for every memory pool.
Uncompiled sketch, following the
MP_raise_release_thresholdpattern:The helper takes an array, so #3415 could pass one entry per device to the same call.