MissingBackpressureException: Could not emit buffer due to lack of requests

问题

I've received a bug report including MissingBackpressureException: Could not emit buffer due to lack of requests for an RxJava Flowable, but I'm struggling to create a simple test case the demonstrates the problem (maintaining the structure of the Flowable).

Here's the test I'm trying to put together, which maintains the same stages in the pipeline:

int inputEvents=10000;

CountDownLatch completed = new CountDownLatch(1);
Flowable<List<String>> flowable = Flowable.<String>create(e -> {

    System.out.println(Thread.currentThread().getName() + ": Will send");
    for (int counter = 0; counter < inputEvents; counter++) {
        e.onNext("" + counter);
        Thread.sleep(5);
    }
    System.out.println(Thread.currentThread().getName() + ": Completed sending");
    e.onComplete();
}, BackpressureStrategy.DROP)
    .onBackpressureDrop(s -> System.out.println("Backpressure, dropping " + Arrays.asList(s)))
    .buffer(1, TimeUnit.SECONDS)
    .doOnNext(strings -> System.out.println("\t" + Thread.currentThread().getName() + ": Buffered: " + strings.size() + " items"))
    .observeOn(Schedulers.io(), false)
    .doOnNext(strings -> {
        System.out.println("\t" + "\t" + Thread.currentThread().getName() + ": Waiting: " + strings.size());
        Thread.sleep(5000);
    });

flowable
    .subscribe(s -> System.out.println("\t" + "\t" + "onNext: " + s.size()),
            error -> {
                throw new RuntimeException(error);
            },
            () -> {
                System.out.println("\t" + "\t" + "Complete");
                completed.countDown();
            });

completed.await();

In production, we get MissingBackpressureException: Could not emit buffer due to lack of requests with the following stack trace:

io.reactivex.rxjava3.exceptions.MissingBackpressureException: Could not emit buffer due to lack of requests
        at io.reactivex.rxjava3.internal.subscribers.QueueDrainSubscriber.fastPathEmitMax(QueueDrainSubscriber.java:87)
        at io.reactivex.rxjava3.internal.operators.flowable.FlowableBufferTimed$BufferExactUnboundedSubscriber.run(FlowableBufferTimed.java:207)
        at io.reactivex.rxjava3.internal.schedulers.ScheduledDirectPeriodicTask.run(ScheduledDirectPeriodicTask.java:39)
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
        at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

So I think this relates to downstream work from the buffer.

However, no matter how long I block in the doOnNext I can't reproduce the problem. Example output:

main: Will send
    RxComputationThreadPool-1: Buffered: 197 items
        RxCachedThreadScheduler-1: Waiting: 197
    RxComputationThreadPool-1: Buffered: 196 items
    RxComputationThreadPool-1: Buffered: 197 items
    RxComputationThreadPool-1: Buffered: 197 items
    RxComputationThreadPool-1: Buffered: 196 items
    RxComputationThreadPool-1: Buffered: 197 items
        onNext: 197
        RxCachedThreadScheduler-1: Waiting: 196
    RxComputationThreadPool-1: Buffered: 197 items
    RxComputationThreadPool-1: Buffered: 197 items
    RxComputationThreadPool-1: Buffered: 196 items
    RxComputationThreadPool-1: Buffered: 197 items
    RxComputationThreadPool-1: Buffered: 197 items
        onNext: 196
        RxCachedThreadScheduler-1: Waiting: 197
    RxComputationThreadPool-1: Buffered: 197 items
    RxComputationThreadPool-1: Buffered: 197 items
...

I was expecting, as the Thread.sleep(5000) takes so long, we would get back pressure.

Is there a way of simulating this, ideally in a test using TestScheduler/TestSubscriber (to avoid the Thread.sleep()s)?

回答1:

I was able to reproduce the MissingBackpressureException by increasing the rate at which your events are emitted, increasing the max number of events, and reducing the rate at which the consumer processes them.

The buffer that's overflowing is the default observeOn(...) operator's buffer of size 128. Since it's receiving a new list once a second, it will take at least a couple of minutes of back pressure before it will overflow.

Note, you can override this default buffer size by passing it as an arg to observeOn(...).

Getting back to backpressure handling, I think the main issue with your pipeline is the buffer(1, TimeUnit.SECONDS) operator. If you look at the javadoc:

Backpressure:This operator does not support backpressure as it uses time. It requests Long.MAX_VALUE upstream and does not obey downstream requests.

As a result of the above your onBackPressureDrop(...) never gets invoked. I think you fix this by placing onBackPressureDrop(...) after buffer(...). Doing so results in your Backpressure, dropping... message.

You should be able to unit test this using: TestScheduler.advanceTimeBy(long, TimeUnit). Though I have to admit, I haven't tried it yet.

来源：https://stackoverflow.com/questions/60756850/missingbackpressureexception-could-not-emit-buffer-due-to-lack-of-requests

标签

rx-java