how to efficiently apply a medium-weight function in parallel

Question

I\'m looking to map a modestly-expensive function onto a large lazy seq in parallel. pmap is great but i\'m loosing to much to context switching. I think I need

Answer 1

I'd look at the ppmap function from: http://www.braveclojure.com/zombie-metaphysics/. It lets you pmap while specifying the chunk size.

The solution to this problem is to increase the grain size, or the amount of work done by each parallelized task. In this case, the task is to apply the mapping function to one element of the collection. Grain size isn’t measured in any standard unit, but you’d say that the grain size of pmap is one by default. Increasing the grain size to two would mean that you’re applying the mapping function to two elements instead of one, so the thread that the task is on is doing more work. [...] Just for fun, we can generalize this technique into a function called ppmap, for partitioned pmap. It can receive more than one collection, just like map:

(defn ppmap
  "Partitioned pmap, for grouping map ops together to make parallel
  overhead worthwhile"
  [grain-size f & colls]
  (apply concat
   (apply pmap
          (fn [& pgroups] (doall (apply map f pgroups)))
          (map (partial partition-all grain-size) colls))))
(time (dorun (ppmap 1000 clojure.string/lower-case orc-name-abbrevs)))
; => "Elapsed time: 44.902 msecs"