Python ZeroMQ PUSH/PULL — Lost Messages?

Question

I am trying to use python with zeroMQ in PUSH / PULL mode, sending messages of size 4[MB] every few seconds.

For

Answer 1

It's conceivable that you are running out of memory (depending how you are sending messages, whether they are being consumed fast enough, etc). You can use socket.setsockopt(zmq.HWM) to set HWM to a sane value and prevent zeromq from storing too many messages in the outgoing buffer. With this in mind, consider slightly modified examples:

# server
...
counter = 0
while True:
    ...receive the message
    counter += 1
    print "Total messages recieved: {0}".format(counter)

# client
socket.setsockopt(zmq.HWM, 8)
for i in range(1000):
    socket.send(msgToSend)

And then running 10 test clients:

for i in {1..10}; do
    python client.py &
done

From the server you can see all the messages are recieved:

Total messages recieved: 9998
Total messages recieved: 9999
Total messages recieved: 10000