shared-memory

I've spent some time investigating memory mapped IO for an application I'm working on. I have some very large (TB scale) files, and I want to map segments from them into memory, for both reading and writing, making maximum use of OS-level caching. The software I'm writing needs to work under Unix/Linux and Windows... performance is critical. I've discovered boost::iostreams::mapped_file_source and boost::iostreams::mapped_file_sink , which provide most of the facilities I'm looking for. The facilities I'd like, but haven't found are: Forcing a synchronisation of written data to disk ( msync (2

I'm trying to dig a bit deeper into parallelziation of R routines. What are my options with respect to the communication of a bunch of "worker" processes regarding the communication between the respective workers ? the communication of the workers with the " master " process? AFAIU, there's no such thing as a " shared environment/shared memory " that both the master as well as all worker processes have access to, right? The best idea I came up with so far is to base the communication on reading and writing JSON documents to the hard drive. That's probably a bad idea ;-) I chose .json over

Problem I'm using Python's multiprocessing module to execute functions asynchronously. What I want to do is be able to track the overall progress of my script as each process calls and executes def add_print . For instance, I would like the code below to add 1 to total and print out the value ( 1 2 3 ... 18 19 20 ) every time a process runs that function. My first attempt was to use a global variable but this didn't work. Since the function is being called asynchronously, each process reads total as 0 to start off, and adds 1 independently of other processes. So the output is 20 1 's instead

I have a multi-process python application (processes are spawned by uwsgi) that needs to store variables in RAM, then read and update those variables from several different processes. I know there are a lot caching options available, but all the ones I've found can only store strings. Is it possible for different python processes to access the same virtual memory, and thus share data without ever converting it or even copying it? Besides POSH, Python Shared Objects, which at least does a part of what you want to do (place Python Objects in SvsV-IPC shared memory and modify them from multiple

I've learned about Python multiprocess's Pipes/Queues/Shared ctypes Objects/Managers, and I want to compare them with Linux's anonymous pipes, named pipes, shared memory, socket, and so on. I now have the following questions The pipes and queue modules of Python's multiprocessing are based on anonymous pipes. Does it provide named pipes? Does Python multiprocessing.sharedctypes support independent process communication? I think it only supports father and child process or brotherly process communication. Which of them are only used in the process of paternity or brotherhood, which can be

We have a need for multiple programs to call functions in a common library. The library functions access and update a common global memory. Each program’s function calls need to see this common global memory. That is one function call needs to see the updates of any prior function call even if called from another program. For compatibility reasons we have several design constraints on how the functions exposed by the shared library must operate: Any data items (both standard data types and objects) that are declared globally must be visible to all callers regardless of the thread in which the

Boost provides a sample atomically reference counted shared pointer Here is the relevant code snippet and the explanation for the various orderings used: class X { public: typedef boost::intrusive_ptr<X> pointer; X() : refcount_(0) {} private: mutable boost::atomic<int> refcount_; friend void intrusive_ptr_add_ref(const X * x) { x->refcount_.fetch_add(1, boost::memory_order_relaxed); } friend void intrusive_ptr_release(const X * x) { if (x->refcount_.fetch_sub(1, boost::memory_order_release) == 1) { boost::atomic_thread_fence(boost::memory_order_acquire); delete x; } } }; Increasing the

I am attempting to stream data from a c++ application to a C# application using shared memory. Based on example I found, I have: c++ (sending) struct Pair { int length; float data[3]; }; #include <windows.h> #include <stdio.h> struct Pair* p; HANDLE handle; float dataSend[3]{ 22,33,44 }; bool startShare() { try { handle = CreateFileMappingW(INVALID_HANDLE_VALUE, NULL, PAGE_READWRITE, 0, sizeof(Pair), L"DataSend"); p = (struct Pair*) MapViewOfFile(handle, FILE_MAP_READ | FILE_MAP_WRITE, 0, 0, sizeof(Pair)); return true; } catch(...) { return false; } } int main() { if (startShare() == true) {

When using shared memory, why should we care about creating a key key_t ftok(const char *path, int id); in the following bit of code? key_t key; int shmid; key = ftok("/home/beej/somefile3", 'R'); shmid = shmget(key, 1024, 0644 | IPC_CREAT); From what I've come to understand, what is needed to access a given shared memory is the shmid , not the key. Or am I wrong? If what we need is the shmid , what is the point in not just creating a random key every time? Edit @ Beej's Guide to Unix IPC one can read: What about this key nonsense? How do we create one? Well, since the type key_t is actually