How to share numpy random state of a parent process with child processes?

Question

I set numpy random seed at the beginning of my program. During the program execution I run a function multiple times using multiprocessing.Process. The function

Answer 1

Fortunately, according to the documentation, you can access the complete state of the numpy random number generator using get_state and set it again using set_state. The generator itself uses the Mersenne Twister algorithm (see the RandomState part of the documentation).

This means you can do anything you want, though whether it will be good and efficient is a different question entirely. As abarnert points out, no matter how you share the parent's state—this could use Alex Hall's method, which looks correct—your sequencing within each child will depend on the order in which each child draws random numbers from the MT state machine.

It would perhaps be better to build a large pool of pseudo-random numbers for each child, saving the start state of the entire generator once at the start. Then each child can draw a PRNG value until its particular pool runs out, after which you have the child coordinate with the parent for the next pool. The parent enumerates which children got which "pool'th" number. The code would look something like this (note that it would make sense to turn this into an infinite generator with a next method):

class PrngPool(object):
    def __init__(self, child_id, shared_state):
        self._child_id = child_id
        self._shared_state = shared_state
        self._numbers = []

    def next_number(self):
        if not self.numbers:
            self._refill()
        return self.numbers.pop(0)  # XXX inefficient

    def _refill(self):
        # ... something like Alex Hall's lock/gen/unlock,
        # but fill up self._numbers with the next 1000 (or
        # however many) numbers after adding our ID and
        # the index "n" of which n-through-n+999 numbers
        # we took here.  Any other child also doing a
        # _refill will wait for the lock and get an updated
        # index n -- eg, if we got numbers 3000 to 3999,
        # the next child will get numbers 4000 to 4999.

This way there is not nearly as much communication through Manager items (MT state and our ID-and-index added to the "used" list). At the end of the process, it's possible to see which children used which PRNG values, and to re-generate those PRNG values if needed (remember to record the full MT internal start state!).

Edit to add: The way to think about this is like this: the MT is not actually random. It is periodic with a very long period. When you use any such RNG, your seed is simply a starting point within the period. To get repeatability you must use non-random numbers, such as a set from a book. There is a (virtual) book with every number that comes out of the MT generator. We're going to write down which page(s) of this book we used for each group of computations, so that we can re-open the book to those pages later and re-do the same computations.