How to produce a lazy sequence by portion in clojure?

Question

I have a database server and I fetch data from it. Sometimes data have millions rows and more so I use laziness for downloading. I use Server Side Cursors from clojure.jdbc library https://funcool.github.io/clojure.jdbc/latest/#cursor-queries to fetch data lazily.

Now I have a problem. I need produce initial 500 elements from a lazy-sequence, then the program must wait for 10 minutes to get a signal which report to the program produce next 500 elements and so on until i receive all data from server. But if a report didn`t arrive for 10 minutes, the program must close a connection.

I wrote sample:

(def lazyseq_maps (atom {:seq_1 {:next_500 false :data nil} :seq_2 {:next_500 false :data nil}})) ; here is a collection of all unfinished lazy sequences waiting for signal to continue produce elements

(jdbc/atomic conn
 (with-open [cursor (jdbc/fetch-lazy conn sql]
   (let [lazyseq (jdbc/cursor->lazyseq cursor)]
     (swap! lazyseq_maps assoc seq_id {:next_500 true :data nil})
     (loop [lazyseq_rest lazyseq
            count 1]
            (if (:next_500 (seq_id @lazyseq_maps))
              (do
                (swap! lazyseq_maps update-in [seq_id :data] conj (first lazyseq_rest))
                (when (= 0 (mod count 500))
                  (swap! lazyseq_maps assoc-in [seq_id :next_500] false))
                (recur (rest lazyseq) (inc count)))
              ;
              (func-for-waiting-signal)))) ; here I don`t know how to create function waiting signal to continue fetching data
   (seq_id @lazyseq_maps)))

Can you help what clojure tools I should use to solve my problems? I assume I should use core.async to create channels for loops. Am I right? And how should I create function which stop executing loop for 10 minutes or continue executing if I receive appropriate signal?

Answer 1

Indeed you should use core.async for this, they have timeout channel for the timeout purpose and with alt! you can "wait" for a value to pop out from any number of channels. The complete code will be a bit complex though. Output rows would be "pushed" to an output channel.

To my knowledge with-open and lazy sequences don't compose nicely, as the cursor would be closed prematurely. But I am not familiar with clojure.jdbc library.