Can I use functions imported from .py files in Dask/Distributed?

Question

I have a question about serialization and imports.

• should functions have their own imports? like I\'ve seen done with PySpark
• Is the following just p

Answer 1

To run an imported function on your cluster that is not available on the workers' environment, you can also create a local function from the imported function. This local function will then be pickled by cloudpickle. In Python 2 you can achieve this with new.function (see the new module). For Python 3 this could be achieved with the types module, but I haven't tried it.

Your example above would then look like:

In [3]: import mod

In [4]: import new

In [5]: def remote(func):
   ...:     return new.function(func.func_code, func.func_globals, closure=func.func_closure)
   ...:

In [6]: e.run(remote(mod.hostname))
Out[6]: {'tcp://10.0.2.15:44208': 'the hostname'}