AsyncFunction - Bug collecting throwable in unorder mode

问题

I am experiencing an infinite loop using an AsyncFunc in unordered mode.

It can be reproduced using the following code

import org.apache.flink.streaming.api.datastream.AsyncDataStream;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.async.AsyncFunction;
import org.junit.Test;

import java.util.Arrays;
import java.util.Collections;
import java.util.concurrent.TimeUnit;

public class AsyncTest {

    @Test
    public void test() throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        DataStream<Integer> withTimestamps = env.fromCollection(Arrays.asList(1,2,3,4,5));

        AsyncDataStream.unorderedWait(withTimestamps, 
            (AsyncFunction<Integer, String>) (input, collector) -> {
                if (input == 3){
                    collector.collect(new RuntimeException("Test"));
                    return;
                }
                collector.collect(Collections.singleton("Ok"));
            }, 10, TimeUnit.MILLISECONDS)
            .returns(String.class)
            .print();

        env.execute("unit-test");
     }
}

My guess would be that the UnorderedStreamElementQueue is the reason of the infinite loop.

It seems to add the StreamElementQueueEntry containing the failing future into its firstSet field but never removes it (as this failing future is not triggering the onCompleteHandler method).

Anyone knows if this could be right or if I am making a mistake in my code?

回答1:

There is indeed a bug in the AsyncWaitOperator. The problem is that it does not react to results which are exceptions (user exceptions as well as timeout exceptions).

In the case of ordered waiting, this causes that exceptional results are only detected if another async result has been completed (without an exception).

In the case of unordered waiting, this causes that the StreamElementQueueEntry is never moved from the firstSet into the completedQueue.

There is an issue for this problem. It will hopefully be fixed in the next couple of days.

来源：https://stackoverflow.com/questions/43639815/asyncfunction-bug-collecting-throwable-in-unorder-mode

标签

apache-flink