[FEA]: VirtualMemoryResource unpickling: drop invisible peers and clarify the invisible-owner error #3419

Is this a duplicate?

• I confirmed there appear to be no duplicate issues for this request and that I agree to the Code of Conduct

Area

cuda.core

Is your feature request related to a problem? Please describe.

Since #3126, a pickled VirtualMemoryResource names its owning device and its peers by UUID, and _reconstruct_virtual_memory_resource maps each UUID back through _device_from_uuid. If any of those devices is not visible to the receiving process, unpickling raises RuntimeError. This includes peers.

A Buffer pickles as (resource, descriptor), so this also affects sending a Buffer through multiprocessing. A common setup fails: a worker pinned to one GPU with CUDA_VISIBLE_DEVICES cannot receive a buffer whose sender listed other GPUs as peers, even though the worker only uses the owning device.

When the error does fire, the message comes from the generic helper:

no CUDA device visible to this process has UUID GPU-...; the visible devices are: 0 (GPU-...). Make the device visible to this process, for example through CUDA_VISIBLE_DEVICES.

It doesn't say that it failed while unpickling a VirtualMemoryResource, or whether the missing UUID is the owning device or a peer.

Describe the solution you'd like

1. Drop peers that aren't visible instead of raising. Peer access only lets other GPUs in the receiving process use that process's mapping. A GPU the receiver can't see can't use it, so granting it access there adds nothing, and the sender's peers keep their access in the sender's process. Visibility can be checked exactly by UUID before anything is imported, so the rebuilt resource can keep only the visible peers and issue a CUDAWarning naming the ones it dropped.
2. Raise a clearer error when the owning device isn't visible. That case still has to raise, because the receiver must map the memory with a resource for the owning device. _reconstruct_virtual_memory_resource should raise an error that says it was unpickling a VirtualMemoryResource and that the missing UUID is the device that owns the memory, instead of passing on the generic _device_from_uuid message.

The VirtualMemoryResource docstring currently says unpickling "raises RuntimeError if one of the devices is not visible" and would need updating. There are no tests for invisible peers yet; this needs tests for both cases, for example on a two-GPU runner with one GPU hidden.

Describe alternatives you've considered

• Keep raising for invisible peers, with a clearer message. This is stricter: a missing peer may mean the processes were misconfigured. In that case the message should name the peer and point to the workaround below.
• Current workaround: send only the descriptor, not the Buffer, and import it with Buffer.from_ipc_descriptor() using a VirtualMemoryResource built in the receiving process for the owning device, with peers that exist there. This works today, but a POSIX file-descriptor descriptor carries no owner UUID, so the receiver must learn the owning device some other way.

Additional context

Follow-up to #3028 and #3126, discussed with @Andy-Jost.