Accessing a single file with multiple threads

Question

I need to access a file concurrently with multiple threads. This needs to be done concurrently, without thread serialisation for performance reasons.

The file in par

Answer 1

Update #2

I wrote some test projects in C to try and figure this out- although Rob Kennedy beat me to the answer while I was away. Both conditions are possible, including cross-process, as he outlines. Here's a link if anyone else would like to see this in action.

SharedFileTests.zip (VS2005 C++ Solution) @ meklarian.com

There are three projects:

InProcessThreadShareTest - Test a creator and client thread.
InProcessThreadShareTest.cpp Snippet @ gist.github

SharedFileHost - Create a host that runs for 1 minute and updates a file.
SharedFileClient - Create a client that runs for 30 seconds and polls a file.
SharedFileHost.cpp and SharedFileClient.cpp Snippet @ gist.github

All of these projects assume the location C:\data\tmp\sharetest.txt is creatable and writable.

---

Update

Given your scenario, sounds like you need a very large chunk of memory. Instead of gaming the system cache, you can use AWE to have access to more than 4Gb of memory, although you will need to map portions at a time. This should cover your L2 scenario as you wish to ensure that physical memory is used.

Address Windowing Extensions @ MSDN

Use AllocateUserPhysicalPages and VirtualAlloc to reserve memory.

AllocateUserPhysicalPages Function (Windows) @ MSDN
VirtualAlloc Function (Windows) @ MSDN

---

Initial

Given that you are using the flag FILE_FLAG_DELETE_ON_CLOSE, is there any reason you wouldn't consider using a memory-mapped file instead?

Managing Memory-Mapped files in Win32 @ MSDN

From what I see in your CreateFile statements, it appears that you want to share data across-thread or across-process, with regard only to having the same file present while any sessions are open. A memory mapped file allows you to use the same logical filename in all sessions. Another benefit is that you can map views and lock portions of the mapped file with safety across all sessions. If you have a strict server with N-client scenario, it should be easy to implement. If you have a case where any client may be the opening server, you may wish to consider using some other mechanism to ensure that only one client gets to initiate the serving file first (via a global mutex, perhaps).

CreateMutex @ MSDN

If you only need one-way transmission of data, perhaps you could use named pipes instead.
(edit) This is best for 1 server to 1 client.

Named Pipes (Windows) @ MSDN