zeromq performance test. What's the accurate latency?

Question

I'm using zmq to carry message across process, and I want to do some performance test to get the latency and throughout.

The official site gives the guide to tell How to Run Performance Tests

For example, I tried:

local_lat tcp://*:15213 200 100000
remote_lat tcp://127.0.0.1:15213 200 100000

and get the result:

message size: 200 [B]
roundtrip count: 100000
average latency: 13.845 [us]

But when trying the pub-sub example in C++, I found the time interval between sending and receiving is about 150us. (I get the result by print log with timestamp)

Could anybody explain the difference between these two?

EDIT: I found the question 0mq: pubsub latency continually growing with messages? The result give a nearly constant delay of 0.00015s, which is 150us, same as my test, 10x than the official performance test. Why is the difference?

Answer 1

I'm having the same problem: ZeroMQ - pub / sub latency

I ran wireshark on my example code which publishes a zeromq message every second. Here is the output of wireshark:

145  10.900249     10.0.1.6 -> 10.0.1.6     TCP 89 5557→51723 [PSH, ACK] Seq=158 Ack=95 Win=408192 Len=33 TSval=502262367 TSecr=502261368
146  10.900294     10.0.1.6 -> 10.0.1.6     TCP 56 51723→5557 [ACK] Seq=95 Ack=191 Win=408096 Len=0 TSval=502262367 TSecr=502262367
147  11.901993     10.0.1.6 -> 10.0.1.6     TCP 89 5557→51723 [PSH, ACK] Seq=191 Ack=95 Win=408192 Len=33 TSval=502263367 TSecr=502262367
148  11.902041     10.0.1.6 -> 10.0.1.6     TCP 56 51723→5557 [ACK] Seq=95 Ack=224 Win=408064 Len=0 TSval=502263367 TSecr=502263367

As you can see it's taking about 45 microseconds to send and acknowledge each message. At first I thought that the connection was getting re-established on each message but that's not it. So I turned my attention to the receiver...

while(true)
    if(subscriver.recv(&message, ZMQ_NOBLOCK)) {
        // print time
    }
}

By adding the ZMQ_NOBLOCK and polling in a hard while loop I got the time down to 100us. That still seems large and it comes at the price of spiking one core. But I do feel like I understand the problem slightly better. Any insight would be appreciated.