F# sequence operations majorly slow compared to List?

问题

Used F# List and Seq to merge two sorted lists/sequences. The values are obtained by reading in two files from secondary memory - the results of the file reads are stored in two sequences. Assuming integers are stored for testing purposes, now trying to merge these to print out a sorted series using this code:

let rec printSortedSeq l1 l2 = 
    match ( l1, l2) with
    | l1,l2 when Seq.isEmpty l1 && Seq.isEmpty l2 -> printfn "";
    | l1, l2 when Seq.isEmpty l1 -> printf "%d " (Seq.head l2);  printSortedSeq l1 (Seq.skip 1 l2);
    | l1, l2 when Seq.isEmpty l2-> printf "%d " (Seq.head l1);  printSortedSeq (Seq.skip 1 l1) [];

    | l1,l2 -> if Seq.head l1 = Seq.head l2 then printf "%d " (Seq.head l1);  printSortedSeq (Seq.skip 1 l1) (Seq.skip 1 l2); 
                               elif Seq.head l1 < Seq.head l2 then printf "%d " (Seq.head l1);  printSortedSeq (Seq.skip 1 l1) (Seq.skip 1 l2); 
                               else printf "%d " (Seq.head l2);  printSortedSeq (Seq.skip 1 l1) (Seq.skip 1 l2);

The code was originally written to merge two sorted lists:

let rec printSortedList l1 l2 = 
    match ( l1, l2) with
    | h1 :: t1 , h2 :: t2 -> if h1 = h2 then printf "%d " h1;  printSortedList t1 t2; 
                               elif h1 < h2 then printf "%d " h1;  printSortedList t1 l2; 
                               else printf "%d " h2;  printSortedList l1 t2;
    | [] , h2 :: t2 ->  printf "%d " h2;  printSortedList [] t2;
    | h1 :: t1, [] -> printf "%d " h1;  printSortedList t1 [];
    | [], [] -> printfn"";

The performance of using them compared hugely in favor of Lists. I'm giving the timing results after doing #time;; in the FSI on some trial inputs.

let x = [0..2..500];
let y = [1..2..100];

let a = {0..2..500}
let b = {1..2..100}

printSortedList x y;; Real: 00:00:00.012, CPU: 00:00:00.015

printSortedSeq a b;; Real: 00:00:00.504, CPU: 00:00:00.515

The question is - is there any way to make things faster using sequences? Because though lists are much faster, since the files that will provide the input are very large ( > 2 GB) they won't fit in main memory and so I am reading in the values from file as a lazy sequence. Converting those to lists before merging kinda defeats the whole purpose.

回答1:

Seq.skip is an anti-pattern. Use LazyList from the F# PowerPack, or use enumerators (GetEnumerator...MoveNext...Current) to efficiently traverse a Seq. See other similar Q&A.

回答2:

As toyvo mentioned, this can be greatly simplified using a stateful enumerator:

let mkStatefulEnum (e: IEnumerator<'T>) =
  let x = ref None
  fun move ->
    if move then x := (if e.MoveNext() then Some e.Current else None)
    !x

let merge (a: seq<'T>) (b: seq<'T>) =
  seq {
    use x = a.GetEnumerator()
    use y = b.GetEnumerator()
    let nextX = mkStatefulEnum x
    let nextY = mkStatefulEnum y
    yield! Seq.unfold (fun (a, b) ->
      match a, b with
      | Some a, Some b -> 
        if a < b then Some (a, (nextX true, nextY false))
        else Some (b, (nextX false, nextY true))
      | Some a, None -> Some (a, (nextX true, nextY false))
      | None, Some b -> Some (b, (nextX false, nextY true))
      | None, None -> None
    ) (nextX true, nextY true)
  }

回答3:

The answer to your question, are F# sequence operations majorly slow compared to List, is no. Your sequence code runs in polynomial time due to sequence re-traversal, while your list code runs in linear time.

For the record, it is possible to merge two sorted sequences in linear time. For example:

open System.Collections.Generic

type State<'T> =
    | Neutral
    | Left of 'T
    | Right of 'T
    | Tail

let mergeSeqs (a: seq<'T>) (b: seq<'T>) =
    let cmp x y =
        match compare x y with
        | 1 -> Some (y, Left x)
        | _ -> Some (x, Right y)
    seq {
        use x = a.GetEnumerator()
        use y = b.GetEnumerator()
        let step st =
            match st with
            | Neutral ->
                match x.MoveNext(), y.MoveNext() with
                | true, true -> cmp x.Current y.Current
                | true, false -> Some (x.Current, Tail)
                | false, true -> Some (y.Current, Tail)
                | false, false -> None
            | Left v ->
                match y.MoveNext() with
                | true -> cmp v y.Current
                | false -> Some (v, Neutral)
            | Right v ->
                match x.MoveNext() with
                | true -> cmp x.Current v
                | false -> Some (v, Neutral)
            | Tail ->
                match x.MoveNext(), y.MoveNext() with
                | false, false -> None
                | true, _ -> Some (x.Current, Tail)
                | _, true -> Some (y.Current, Tail)
        yield! Seq.unfold step Neutral
    }

You can improve on that by reducing consing. Design a custom IEnumerator with mutable state similar to State<'T>, and use that as the basis for the merged sequence.

来源：https://stackoverflow.com/questions/11201850/f-sequence-operations-majorly-slow-compared-to-list

标签

list

Sequence