Better alternative to pmap in Clojure for parallelizing moderately inexpensive functions over big data?

Question

Using clojure I have a very large amount of data in a sequence and I want to process it in parallel, with a relatively small number of cores (4 to 8).

The ea

Answer 1

Sadly not a valid answer yet, but something to watch for in the future is Rich's work with the fork/join library coming in Java 7. If you look at his Par branch on github he's done some work with it, and last I had seen the early returns were amazing.

Example of Rich trying it out.

http://paste.lisp.org/display/84027

Answer 2

This question: how-to-efficiently-apply-a-medium-weight-function-in-parallel also addresses this problem in a very similar context.

The current best answer is to use partition to break it into chunks. then pmap a map function onto each chunk. then recombine the results. map-reduce-style.

Answer 3

The fork/join work mentioned in earlier answers on this and similar threads eventually bore fruit as the reducers library, which is probably worth a look.

Answer 4

You can use some sort of map/reduce implemented by hand. Also take a look at swarmiji framework.

"A distributed computing system that helps writing and running Clojure code in parallel - across cores and processors"