clEnqueueNDRange blocking on Nvidia hardware? (Also Multi-GPU)

Question

On Nvidia GPUs, when I call clEnqueueNDRange, the program waits for it to finish before continuing. More precisely, I'm calling its equivalent C++ binding, CommandQueue::enqueueNDRange, but this shouldn't make a difference. This only happens on Nvidia hardware (3 Tesla M2090s) remotely; on our office workstations with AMD GPUs, the call is nonblocking and returns immediately. I don't have local Nvidia hardware to test on - we used to, and I remember similar behavior then, too, but it's a bit hazy.

This makes spreading the work across multiple GPUs harder. I've tried starting a new thread for each call to enqueueNDRange using std::async/std::finish in the new C++11 spec, but that doesn't seem to work either - monitoring the GPU usage in nvidia-smi, I can see that the memory usage on GPU 0 goes up, then it does some work, then the memory on GPU 0 goes down and the memory on GPU 1 goes up, that one does some work, etc. My gcc version is 4.7.0.

Here's how I'm starting the kernels, where increment is the desired global work size divided by the number of devices, rounded up to the nearest multiple of the desired local work size:

std::vector<cl::CommandQueue> queues;
/* Population of queues happens somewhere
cl::NDrange offset, increment, local;
std::vector<std::future<cl_int>> enqueueReturns;
int numDevices  = queues.size();

/* Calculation of increment (local is gotten from the function parameters)*/

//Distribute the job among each of the devices in the context
for(int i = 0; i < numDevices; i++)
{   
    //Update the offset for the current device
    offset = cl::NDRange(i*increment[0], i*increment[1], i*increment[2]);

    //Start a new thread for each call to enqueueNDRangeKernel
    enqueueReturns.push_back(std::async(
                   std::launch::async,
                   &cl::CommandQueue::enqueueNDRangeKernel,
                   &queues[i],
                   kernels[kernel],
                   offset,
                   increment,
                   local,
                   (const std::vector<cl::Event>*)NULL,
                   (cl::Event*)NULL));
    //Without those last two casts, the program won't even compile
}   
//Wait for all threads to join before returning
for(int i = 0; i < numDevices; i++)
{   
    execError = enqueueReturns[i].get();

    if(execError != CL_SUCCESS)
        std::cerr << "Informative error omitted due to length" << std::endl
}   

The kernels definitely should be running on the call to std::async, since I can create a little dummy function, set a breakpoint on it in GDB and have it step into it the moment std::async is called. However, if I make a wrapper function for enqueueNDRangeKernel, run it there, and put in a print statement after the run, I can see that it takes some time between prints.

P.S. The Nvidia dev zone is down due to hackers and such, so I haven't been able to post the question there.

EDIT: Forgot to mention - The buffer that I'm passing to the kernel as an argment (and the one I mention, above, that seems to get passed between the GPUs) is declared as using CL_MEM_COPY_HOST_PTR. I had been using CL_READ_WRITE_BUFFER, with the same effect happening.

Answer 1

I emailed the Nvidia guys and actually got a pretty fair response. There's a sample in the Nvidia SDK that shows, for each device you need to create seperate:

• queues - So you can represent each device and enqueue orders to it
• buffers - One buffer for each array you need to pass to the device, otherwise the devices will pass around a single buffer, waiting for it to become available and effectively serializing everything.
• kernel - I think this one's optional, but it makes specifying arguments a lot easier.

Furthermore, you have to call EnqueueNDRangeKernel for each queue in separate threads. That's not in the SDK sample, but the Nvidia guy confirmed that the calls are blocking.

After doing all this, I achieved concurrency on multiple GPUs. However, there's still a bit of a problem. On to the next question...

Answer 2

Yes, you're right. AFAIK - the nvidia implementation has a synchronous "clEnqueueNDRange". I have noticed this when using my library (Brahma) as well. I don't know if there is a workaround or a way of preventing this, save using a different implementation (and hence device).