Stack overflow when using Retrofit rxjava concatWith

Question

I want to handle pagination in Retrofit using rxjava Observable. I followed the advice from another question.

I have more than 100 pages that needs to be fetched, but chain fails around the 20th page and stops any further subscription to the observable with the below log in the logcat

04-04 04:12:11.766    2951-3012/com.example.app I/dalvikvm﹕ threadid=28: stack overflow on call to Ljava/util/concurrent/atomic/AtomicLongFieldUpdater$CASUpdater;.compareAndSet:ZLJJ
04-04 04:12:11.766    2951-3012/com.example.app I/dalvikvm﹕ method requires 56+20+32=108 bytes, fp is 0x94b52350 (80 left)
04-04 04:12:11.766    2951-3012/com.example.app I/dalvikvm﹕ expanding stack end (0x94b52300 to 0x94b52000)
04-04 04:12:11.766    2951-3012/com.example.app I/dalvikvm﹕ Shrank stack (to 0x94b52300, curFrame is 0x94b548dc)

Does anybody know why this might happen?

Update: I know this happens due to recursion, but is there a more graceful way of handling pagination with retrofit and rxjava?

Answer 1

So, given I answered the original question you referenced I should probably try to answer this case as well. :)

This is another valid (and potentially simpler) alternative to my original paging answer, now that I've developed a few more Rx tricks in my arsenal. :) (Done in java8 lambda-style pseudocode):

Observable.range(Integer.MAX_VALUE)
    // Get each page in order.
    .concatMap(page -> getResults(page))
    // Take every result up to and including the one where the next page index is null.
    .takeUntil(result -> result.next == null)
    // Add each output to a list builder. I'm using Guava's ImmutableList, but you could
    // just as easily use a regular ArrayList and avoid having to map afterwards. I just
    // personally prefer outputting an immutable data structure, and using the builder
    // is natural for this.
    //
    // Also, if you wanted to have the observable stream the full output at each page,
    // you could use collect instead of reduce. Note it has a slightly different syntax. 
    .reduce(
        ImmutableList.<ResponseObject>builder(),
        (builder, response) -> builder.addAll(response.results))
    // Convert list builder to one List<ResponseObject> of all the things.
    .map(builder -> builder.build())
    .subscribe(results -> { /* Do something with results. */ });