Inconsistency of IDs between 'nvidia-smi -L' and cuDeviceGetName()

Question

I\'m running this command into a shell and get:

C:\\Users\\me>nvidia-smi -L    
GPU 0: Quadro K2000 (UUID: GPU-b1ac50d1-019c-58e1-3598-4877fffffd3f17)    
G         


        

Answer 1

You can set the device order for CUDA environment in your shell to follow the bus ID instead of the default of fastest card. Requires CUDA 7 and up.

export CUDA_DEVICE_ORDER=PCI_BUS_ID

Answer 2

It's the expected behaviour.

nvidia-smi manpage says that

the GPU/Unit's 0-based index in the natural enumeration returned by the driver,

CUDA API enumerates in descending order of compute capability according to "Programming Guide" 3.2.6.1 Device enumeration.

I had this problem and I have written a program is analog of nvidia-smi, but with enumerated devices in an order consistent with CUDA API. Farther in the text ref on the program

https://github.com/smilart/nvidia-cdl

I have written the program because nvidia-smi cannot enumerated device in an order consistent with CUDA API.

Answer 3

It's expected behavior.

nvidia-smi enumerates in PCI order.

By default, the CUDA driver and runtime APIs do not.

The question you linked clearly shows how to associate the two numbering/ordering schemes.

There is no way to cause nvidia-smi to modify its ordering scheme to match whatever will be generated by the CUDA runtime or driver APIs. However you can modify the CUDA runtime enumeration order through the use of an environment variable in CUDA 8.