Based on the documentation, the default GPU is the one with the lowest id:
If you have more than one GPU in your system, the GPU with the lowest ID
Just to be clear regarding the use of the environment variable CUDA_VISIBLE_DEVICES:
To run a script my_script.py on GPU 1 only, in the Linux terminal you can use the following command:
username@server:/scratch/coding/src$ CUDA_VISIBLE_DEVICES=1 python my_script.py
More examples illustrating the syntax:
Environment Variable Syntax Results
CUDA_VISIBLE_DEVICES=1 Only device 1 will be seen
CUDA_VISIBLE_DEVICES=0,1 Devices 0 and 1 will be visible
CUDA_VISIBLE_DEVICES="0,1" Same as above, quotation marks are optional
CUDA_VISIBLE_DEVICES=0,2,3 Devices 0, 2, 3 will be visible; device 1 is masked
CUDA_VISIBLE_DEVICES="" No GPU will be visible
FYI: