lmax disruptor is too slow in multi-producer mode compared to single-producer mode

问题

Previously, when I use single-producer mode of disruptor, e.g.

 new Disruptor<ValueEvent>(ValueEvent.EVENT_FACTORY,
        2048, moranContext.getThreadPoolExecutor(), ProducerType.Single,
        new BlockingWaitStrategy())

the performance is good. Now I am in a situation that multiple threads would write to a single ring buffer. What I found is that ProducerType.Multi make the code several times slower than single producer mode. That poor performance is not going to be accepted by me. So should I use single producer mode while multiple threads invoke the same event publish method with locks, is that OK? Thanks.

回答1:

IMHO, single producer accessed by multi threads with lock won't resolve your problem, because it simply shift the locking from the disruptor side to your own program.

The solution to your problem varies from the type of event model you need. I.e. do you need the events to be consumed chronologically; merged; or any special requirement. Since you are dealing with disruptor and multi producers, that sounds to me very much like FX trading systems :-) Anyway, based on my experience, assuming you need chronological order per producer but don't care about mixing events between producers, I would recommend you to do a queue merging thread. The structure is

Each producer produces data and put them into its own named queue
A worker thread constantly examine the queues. For each queue it remove one or several items and put it to the single producer of your single producer disruptor.

Note that in the above scenario,

Each producer queue is a single producer single consumer queue.
The disruptor is a single producer multi consumer disruptor.
Depends on your need, to avoid a forever running thread, if the thread examine for, say, 100 runs and all queues are empty, it can set some variable and go wait() and the event producers can yield() it when seeing it's waiting.

I think this resolve your problem. If not please post your need of event processing pattern and let's see.

回答2:

I'm somewhat new to the Disruptor, but after extensive testing and experimenting, I can say that ProducerType.MULTI is more accurate and faster for 2 or more producer threads.

With 14 producer threads on a MacBook, ProducerType.SINGLE shows more events published than consumed, even though my test code is waiting for all producers to end (which they do after a 10s run), and then waiting for the disruptor to end. Not very accurate: Where do those additional published events go?

Driver start: PID=38619 Processors=8 RingBufferSize=1024 Connections=Reuse Publishers=14[SINGLE] Handlers=1[BLOCK] HandlerType=EventHandler<Event>
  Done: elpased=10s eventsPublished=6956894 eventsProcessed=4954645
  Stats: events/sec=494883.36 sec/event=0.0000 CPU=82.4%

Using ProducerType.MULTI, fewer events are published than with SINGLE, but more events are actually consumed in the same 10 seconds than with SINGLE. And with MULTI, all of the published events are consumed, just what I would expect due to the careful way the driver shuts itself down after the elapsed time expires:

Driver start: PID=38625 Processors=8 RingBufferSize=1024 Connections=Reuse Publishers=14[MULTI] Handlers=1[BLOCK] HandlerType=EventHandler<Event>
  Done: elpased=10s eventsPublished=6397109 eventsProcessed=6397109
  Stats: events/sec=638906.33 sec/event=0.0000 CPU=30.1%

Again: 2 or more producers: Use ProducerType.MULTI.

By the way, each Producer publishes directly to the ring buffer by getting the next slot, updating the event, and then publishing the slot. And the handler gets the event whenever its onEvent method is called. No extra queues. Very simple.

来源：https://stackoverflow.com/questions/18997398/lmax-disruptor-is-too-slow-in-multi-producer-mode-compared-to-single-producer-mo

标签

java

multithreading