问题

I am processing a Subrip subtitles file which is quite large and need to process it one subtitle at a time. In Java, to extract the subtitles from file, I would write a method with following signature:

Iterator<Subtitle> fromSubrip(final Iterator<String> lines);

The use of Iterator gives me two benefits:

The file is never in the memory in its entirety, nor is any of its transformed stage.
An abstraction wherein I can loop over a collection of Subtitle objects without the memory overhead.

Since iterators are by nature imperative and mutable, they're probably not idiomatic in Clojure. So what is the Clojure way to deal with this sort of situation?

回答1:

As Vladimir said, you need to handle the laziness and file closing correctly. Here's how I did it, as shown in "Read a very large text file into a list in clojure":

(defn lazy-file-lines 
  "open a (probably large) file and make it a available as a lazy seq of lines"
  [filename]
  (letfn [(helper [rdr]
                  (lazy-seq
                    (if-let [line (.readLine rdr)]
                      (cons line (helper rdr))
                      (do (.close rdr) nil))))]
         (helper (clojure.java.io/reader filename))))

回答2:

read all files from a directory, a lazy way.

using go black and channel.

code:

(ns user
  (:require [clojure.core.async :as async :refer :all 
:exclude [map into reduce merge partition partition-by take]]))

(defn read-dir [dir]
  (let [directory (clojure.java.io/file dir)
        files (filter #(.isFile %) (file-seq directory))
        ch (chan)]
    (go
      (doseq [file files]
        (with-open [rdr (clojure.java.io/reader file)]
          (doseq [line (line-seq rdr)]
            (>! ch line))))
      (close! ch))
    ch))

invoke:

(def aa "D:\\Users\\input")
(let [ch (read-dir aa)]
  (loop []
    (when-let [line (<!! ch )]
      (println line)
      (recur))))

================

reify the Iterable interace, can be used in java.

MyFiles.clj:
(ns user
  (:gen-class :methods [#^{:static true} [readDir [String] Iterable]])
  (:require [clojure.core.async :as async :refer :all 
:exclude [map into reduce merge partition partition-by take]]))

(defn -readDir [dir]
  (def i nil)
  (let [ch (read-dir dir)
        it (reify java.util.Iterator
             (hasNext [this] (alter-var-root #'i (fn [_] (<!! ch))) (not (nil? i)))
             (next [this] i))
        itab (reify Iterable
               (iterator [this] it))]
    itab))

java code:

for (Object line : MyFiles.readDir("/dir")) {
    println(line)
}

回答3:

You can use lazy sequences for this, for example, line-seq.

You must be careful, however, that the sequence returned by line-seq (and other functions which return lazy sequences based on some external resource) never would leak out of e.g. with-open scope because after the source is closed, further reading from lazy sequence will cause exceptions.

来源：https://stackoverflow.com/questions/15996035/clojure-way-of-reading-large-files-and-transforming-data-therein

标签

file-io