How to prevent two CUDA programs from interfering

Question

I\'ve noticed that if two users try to run CUDA programs at the same time, it tends to lock up either the card or the driver (or both?). We need to either reset the card or

Answer 1

You can use something like Task Spooler to queue the programs and run one at the time.

We use TORQUE Resource Manager but it's harder to configure than ts. With TORQUE you can have multiple queues (ie one for cuda jobs, two for cpu jobs) and assign a different job to each gpu.