Pytorh has an example here on how to train networks on Imagenet. At line 102, we have ngpus_per_node = torch.cuda.device_count() which takes all the GPU\'s to a
ngpus_per_node = torch.cuda.device_count()