multicore

I'm working with mclapply from the multicore package (on Ubuntu), and I'm writing a function that required that the results of mclapply(x, f) are returned in order (that is, f(x[1]), f(x[2]), ...., f(x[n]) ). # multicore doesn't work on Windows require(multicore) unlist(mclapply( 1:10, function(x){ Sys.sleep(sample(1:5, size = 1)) identity(x)}, mc.cores = 2)) [1] 1 2 3 4 5 6 7 8 9 10 The above code seems to imply that mclapply returns results in the same order as lapply . However, if this assumption is wrong I'll have to spend a long time refactoring my code, so I'm hoping to get assurance

I keep hitting an issue with the multicore package and big objects. The basic idea is that I'm using a Bioconductor function ( readBamGappedAlignments ) to read in large objects. I have a character vector of filenames, and I've been using mclapply to loop over the files and read them into a list. The function looks something like this: objects <- mclapply(files, function(x) { on.exit(message(sprintf("Completed: %s", x))) message(sprintf("Started: '%s'", x)) readBamGappedAlignments(x) }, mc.cores=10) However, I keep getting the following error: Error: serialization is too large to store in a

Fairly new to Parallel R. Quick question. I have an algorithm that is computationally intensive. Fortunately it can easily be broken up into pieces to make use of multicore or snow . What I would like to know is if it is considered fine in practice to use multicore in conjunction with snow ? What I would like to do is split up my load to run on multiple machines in a cluster and for each machine. I would like to utilize all cores on the machine. For this type of processing, is it reasonable to mix snow with multicore ? I have used the approach suggested above by lockedoff, that is use the

Recently I stumbled upon http://dask.pydata.org/en/latest/ As I have some pandas code which only runs on a single core I wonder how to make use of my other CPU cores. Would dask work well to use all (local) CPU cores? If yes how compatible is it to pandas? Could I use multiple CPUs with pandas? So far I read about releasing the GIL but that all seems rather complicated. Would dask work well to use all (local) CPU cores? Yes. how compatible is it to pandas? Pretty compatible. Not 100%. You can mix in Pandas and NumPy and even pure Python stuff with Dask if needed. Could I use multiple CPUs with

I'm doing some research on multicore processors; specifically I'm looking at writing code for multicore processors and also compiling code for multicore processors. I'm curious about the major problems in this field that would currently prevent a widespread adoption of programming techniques and practices to fully leverage the power of multicore architectures. I am aware of the following efforts (some of these don't seem directly related to multicore architectures, but seem to have more to do with parallel-programming models, multi-threading, and concurrency): Erlang (I know that Erlang

Amazon measures their CPU allotment in terms of virtual cores and EC2 Compute Units. EC2 Compute Units are defined as: The amount of CPU that is allocated to a particular instance is expressed in terms of these EC2 Compute Units. We use several benchmarks and tests to manage the consistency and predictability of the performance from an EC2 Compute Unit. One EC2 Compute Unit provides the equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor. This is also the equivalent to an early-2006 1.7 GHz Xeon processor referenced in our original documentation. My question is, say I

In my application, threads need to pause for a very little time (100s of clock cycles). One way to pause is to call nanosleep, but I suppose it requires a system call to the kernel. Now I want to pause without going to the kernel. Note that I have enough cores to run my threads on and I bind each thread to a separate core, so even an instruction that can halt the core for a little while would be good. I am using x86. I just want the thread to halt while pausing. I don't want a busy loop or a system call to the kernel. Is it possible to do this? What is the minimum time I can pause a thread?

Closed . This question is opinion-based. It is not currently accepting answers. Learn more . Want to improve this question? Update the question so it can be answered with facts and citations by editing this post . Straightforward C#/Java code is extremely difficult to parallelize, multi-thread, etc. As a result, straightforward C#/Java code will use less and less of the total processing power on a box (because everything is now going to be multi-core). Solving this problem in C# and Java is not simple. Mutability and side effects are key to getting stuff done in C# and Java, but this is