rxJava buffer() with time that honours backpressure

问题

The versions of buffer operator that don't operate on time honour backpressure as per JavaDoc:

http://reactivex.io/RxJava/2.x/javadoc/io/reactivex/Flowable.html#buffer-int-

However, any version of buffer that involves time based buffers doesn't support backpressure, like this one

http://reactivex.io/RxJava/2.x/javadoc/io/reactivex/Flowable.html#buffer-long-java.util.concurrent.TimeUnit-int-

I understand this comes from the fact that once the time is ticking, you can't stop it similarly to, for example interval operator, that doesn't support backpressure either for the same reason.

What I want is a buffering operator that is both size and time based and fully supports backpressure by propagating the backpressure signals to BOTH the upstream AND the time ticking producer, something like this:

someFlowable() .buffer( Flowable.interval(1, SECONDS).onBackpressureDrop(), 10 );

So now I could drop the tick on backpressure signals.

Is this something currently achievable in rxJava2? How about Project-Reactor?

回答1:

I've encountered the problem recently and here is my implementation. It can be used like this:

    Flowable<List<T>> bufferedFlow = (some flowable of T)
                              .compose(new BufferTransformer(1, TimeUnit.MILLISECONDS, 8))

It supports backpressure by the count you've specified.

Here is the implementation: https://gist.github.com/driventokill/c49f86fb0cc182994ef423a70e793a2d

回答2:

I had problems with solution from https://stackoverflow.com/a/55136139/6719538 when used DisposableSubscriber as subscribers, and as far as I can see this transformer don't consider calls Suscription#request from downstream subscribers (it could overflow them). I create my version that was tested in production - BufferTransformerHonorableToBackpressure.java. fang-yang - great respect for idea.

回答3:

It's been a while, but I had a look at this again and somehow it struck me that this:

public static <T> FlowableTransformer<T, List<T>> buffer(
    int n, long period, TimeUnit unit)
{
    return o ->
        o.groupBy(__ -> 1)
         .concatMapMaybe(
             gf ->
                 gf.take(n)
                   .take(period, SECONDS)
                   .toList()
                   .filter(l -> !l.isEmpty())
         );
}

is pretty much doing what I described. That, if I am correct is fully backpressured and will either buffer n items or after specified time if enough items haven't been collected

回答4:

I had another go at it that lead to a quite overengineered solution that seems to be working (TM)

The requirements:

A buffering operator that releases a buffer after a time interval elapses, or the buffer reaches maximum size, whichever happens first
The operator has to be fully backpressured, that is, if requests cease from downstream, the buffer operator should not emit data nor should it raise any exception (like the starndard Flowable.buffer(interval, TimeUnit) operator does. The operator should not consume its source/upstream in an unbounded mode either
Do this with composing existing/implemented operators.

Why would anyone want it?:

The need for such operator came when I wanted to implement a buffering on an infinite/long running stream. I wanted to buffer for efficiency, but the standard Flowable.buffer(n) is not suitable here since an "infinite" stream can emit k < n elements and then not emit items for a long time. Those k elements are trapped in buffer(n). So adding timeout would do the job, but in rxJava2 the buffer operator with timeout doesn't honor backpressure and buffering/dropping or any other built in strategy is not good enough.

The solution outline:

The solution is based on generateAsync and partialCollect operators, both implemented in https://github.com/akarnokd/RxJava2Extensions project. The rest is starndard RxJava2.

First wrap all the values from upstream in a container class C
Then merge that stream with a stream which source is using generateAsync. That stream uses switchMap to emit instancesof C that are effectively timeout signals.
The two merged streams are flowing into partialCollect that holds a reference to an "API" object to emit items into the generateAsync upstream. This is a sort of feedback loop that goes from paritialCollect via the "API" object to generateAsync that feeds back to partialCollect. In this way partialCollect can upon receiving the first element in a buffer emit a signal that will effectively start a timeout. If the buffer doesn't fill before the timeout, it will cause an instance of empty C (not containing any value) flowing back into partialCollect. It will detect it as a timeout signal and release the aggregated buffer downstream. If the buffer is released because of reaching its maximum size, it will be released and the next item will kick off another timeout. Any timeout signal (an instance of empty C) arriving late, aka after the buffer has been released because of reaching maximum size will be ignored. It is possible, because it's the partialCollect that instantiate and sends out the timeout signal item that will potentially flow back to it. Checking the identity of that item allows to detect a late vs legitimate timeout signal.

The code: https://gist.github.com/artur-jablonski/5eb2bb470868d9eeeb3c9ee247110d4a

来源：https://stackoverflow.com/questions/50040510/rxjava-buffer-with-time-that-honours-backpressure

标签

rx-java2

project-reactor