IO completion port key confusion

Question

I'm writing an IO completion port based server (source code here) using the Windows DLL API in Python using the ctypes module. But this is a pretty direct usage of the API and this question is directed at those who have a knowledge of IOCP, not Python.

As I understand the documentation for CreateIoCompletionPort, you specify your "user defined" completion key when you call this function with a file handle (in my case a socket) you are associating with the created IOCP. When you get around to calling GetQueuedCompletionStatus, you get a completion key value along with a pointer to an overlapped object. The completion key should identify what overlapped object and request has completed.

However, let's say I pass in 100 as the completion key in my CreateIoCompletionPort call with an overlapped object. When the same overlapped object has its IO completed and it arrives back through GetQueuedCompletionStatus, the completion key that accompanies it is much larger and bares no resemblance to the original value of 100.

Am I misunderstanding how the completion key works, or must I be doing it wrong in the source code I linked above?

Answer 1

What I've found in everyday practice is that it is best to simply focus on the OVERLAPPED result, as that will be unchanged. One way that you can use it effectively is to have something like the following:

struct CompletionHandler
{
    OVERLAPPED dummy_ovl;
    /* Stuff that actually means something to you here */
};

When you post something to the IOCP (whether via I/O call or just a post via Win32 API), you first create a CompletionHandler object that you will use to track the call, and cast the address of that object to OVERLAPPED*.

CompletionHander my_handler;
// Fill in whatever you need to in my_handler
// Don't forget to keep the original my_handler!

// I/O call goes here, and for OVERLAPPED* give: (OVERLAPPED*)&my_handler

This way, when you get the OVERLAPPED result all you have to do is cast it back to CompletionHandler and voila! You have the original context of your call.

OVERLAPPED* from_queued_completion_status;
// Actually get a value into from_queued_completion_status

CompletionHandler* handler_for_this_completion = (CompletionHandler*)from_queued_completion_status;
// Have fun!

For more details in a real-world setting, check out Boost's implementation of ASIO for Windows (ver 1.42 header here). There are some details like validation the OVERLAPPED pointer you get from GetQueuedCompletionStatus, but again, see the link for a good way to implement.

Answer 2

GetQueuedCompletionStatus returns two things, an OVERLAPPED structure and a completion key. The completion key represents per-device information, and the OVERLAPPED structure represents per-call information. The completion key should match what was given in the call to CreateIoCompletionPort. Typically, you would use a pointer to a structure containing information about the connection as the completion key.

It looks like you're not doing anything with completionKey as returned by GetQueuedCompletionStatus.

I'm guessing you want:

if completionKey != acceptKey:
    Cleanup()
    ...

edit:

Does Python somehow know that the OVERLAPPED structure created in CreateAcceptSocket is being used asynchronously by the Win32 API and prevent it from being GC'd?

Answer 3

You should think about the completion key as 'per connection' data and the (extended) overlapped structure as 'per i/o' operation.

Some people use an extended overlapped structure for BOTH and store all the information that they need in the extended overlapped structure. I have always stored a reference counted object that wraps my socket in the completion key and a reference counted data buffer as an extended overlapped structure. If you're interested you can see some example IOCP code in C++ here.

The completion key is, effectively, just an opaque handle to data that the I/O completion system will give back to you when a completion occurs on the socket.

Answer 4

The problem is in how I am passing the completion keys. The completion key argument is a pointer, yet it passes back the pointer not the value pointed to - a little confusing to me at least.

Also the completion key passed for the accepted connection overlapped packet is that of the listening socket - not the accepted socket.

Answer 5

Completion key is not a pointer - it is a number of type ULONG_PTR, which just means "an integer the size of a pointer" so it is 32-bits on x86 and 64-bits on x64. The typename is confusing, but when win32 typenames refer to a pointer they do it by having a P at the front of the name, not the end.

Answer 6

If your application is multithreading, make sure that the CompletionKey you're passing is either constant or a pointer value to object on heap rather than on stack. In your example where 100 is passed as constant, you must be wrong to say any change. But as to the problem it could be that you pass a socket handle in CreateIoCompletionPort but not a reference to handle in GetQueuedCompletionStatus in order to retrive it. You can do

HANDLE socket;
CreateIoCompletionPort((HANDLE)socket, existed_io_completion_port, (ULONG_PTR)socket, 0);
/*some I/Os*/
...

and

HANDLE socket;
GetQueuedCompletionStatus(existed_io_completion_port, &io_bytes_done, (PULONG_PTR)&socket, &overlapped);

and pay attention to the type in bracket.