问题
Is it possible to run dask
from a python script?
In interactive session I can just write
from dask.distributed import Client
client = Client()
as described in all tutorials. If I write these lines however in a script.py
file and execute it python script.py
, it immediately crashes.
I found another option I found, is to use MPI:
# script.py
from dask_mpi import initialize
initialize()
from dask.distributed import Client
client = Client() # Connect this local process to remote workers
And then run the script with mpirun -n 4 python script.py
. This doesn't crash, however if you print the client
print(client)
# <Client: scheduler='tcp://137.250.37.84:35145' processes=0 cores=0>
you see that no cores are used, accordingly scripts run forever without doing anything.
How do I set my scripts up correctly?
回答1:
If you want to create processes from within a Python script you need to protect that code in an if __name__ == "__main__":
block
from dask.distributed import Client
if __name__ == "__main__":
client = Client()
If you want to use dask-mpi then you need to run it with mpirun
or mpiexec
with a suitable number of processes.
来源:https://stackoverflow.com/questions/57575843/using-dask-from-script