ipython-parallel

问题 I recently made an update of IPython to 4.0.0 and installed Jupyter 4.0.6. I wanted to use Ipython parallel, and after starting the engines in the notebook, I imported: from IPython import parallel And it fails: ~/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/IPython/utils/traitlets.py:5: UserWarning: IPython.utils.traitlets has moved to a top-level traitlets package. warn("IPython.utils.traitlets has moved to a top-level traitlets package.") ~/Library/Enthought/Canopy_64bit

问题 I am running a bunch of long-running tasks with IPython's great parallelization functionality. How can I get real-time output from the ipengines' stdout in my IPython client? E.g., I'm running dview.map_async(fun, lots_of_args) and fun prints to stdout. I would like to see the outputs as they are happening. I know about AsyncResult.display_output() , but it's only available after all tasks have finished. 回答1: You can see stdout in the meantime by accessing AsyncResult.stdout , which will

问题 Since the awesome 1.0.0 release I've been playing around with iPython parallel interface. What I'm trying to do is to set up a asynchronous stochastic gradient descent system. The way I see it, I want to send a function to all the nodes and get the results as they come out. From what I was able to implement and glance from the documentation the standard views implemented don't really support that. The get(timeout) method would do that, but you can't really loop through every entry in a <ASync

问题 Hopefully someone can enlighten me without me having to post a lot of confusing code. I am using IPython.parallel to process neural networks. In an attempt to find a bug I decided to send the same network out to each client with the same input data. I would expect to have each client return the same answer, which most of the time, it does. However sometimes I will get vastly different results from each client. here's just a sample of running the code 5 different times. Each time the code is

问题 I use a SGE cluster with IPcontroller running on the head node, and ~50 engines running on the other nodes (submitted using QSUB). The engines are able to connect and register with the controller without any issues. I can also connect to the head node using SSH and view the engines IDs and running simple code. For e.g., this works perfectly well : %px %pylab inline parallel_result = lbView.map_sync(lambda x: x*rand(), range(32)) However, when I try to run the following line, then the engines

问题 I have successfully configured mpi with mpi4py support across three nodes, as per testing of the hellowworld.py script in the mpi4py demo directory: gms@host:~/development/mpi$ mpiexec -f machinefile -n 10 python ~/development/mpi4py/demo/helloworld.py Hello, World! I am process 3 of 10 on host. Hello, World! I am process 1 of 10 on worker1. Hello, World! I am process 6 of 10 on host. Hello, World! I am process 2 of 10 on worker2. Hello, World! I am process 4 of 10 on worker1. Hello, World! I

问题 Related to this thread... I am trying to track down a bug in which the results from processing on an iPython cluster do not match what happens when the same process is run locally. Even when the iPython cluster is entirely local, and the CPU is simply running multiple engines. I cannot seem to figure out how to log data as it is being processed on the engines. Print statements don't work and even when I try to have each engine write to a separate file, the file is created but nothing is

问题 I'm using Ipython parallel in an optimisation algorithm that loops a large number of times. Parallelism is invoked in the loop using the map method of a LoadBalancedView (twice), a DirectView 's dictionary interface and an invocation of a %px magic. I'm running the algorithm in an Ipython notebook. I find that the memory consumed by both the kernel running the algorithm and one of the controllers increases steadily over time, limiting the number of loops I can execute (since available memory

问题 I originally wrote a nested for loop over a test 3D array in python. As I wanted to apply it to larger array which would take a lot more time, I decided to parallelise using ipyparallel by writing it as a function and using bview.map. This way I could take advantage of multiple cores/nodes on a supercomputer. However the code is actually slower when sent to the supercomputer. When I profiled, it appears the time is spent mostly on method 'acquire' of 'thread.lock' objects which from other

问题 I've been playing around with IPython.parallel and I wanted to use some custom modules of my own, but haven't been able to do it as explained on the cookbook using dview.sync_imports() . The only thing that has worked for me was something like def my_parallel_func(args): import sys sys.path.append('/path/to/my/module') import my_module #and all the rest and then in the main just to if __name__=='__main__': #set up dview... dview.map( my_parallel_func, my_args ) The correct way to do this