dask

问题 This is my first venture into parallel processing and I have been looking into Dask but I am having trouble actually coding it. I have had a look at their examples and documentation and I think dask.delayed will work best. I attempted to wrap my functions with the delayed(function_name), or add an @delayed decorator, but I can't seem to get it working properly. I preferred Dask over other methods since it is made in python and for its (supposed) simplicity. I know dask doesn't work on the for

问题 I have two dataframe dataframe1: >df_case = dd.read_csv('s3://../.../df_case.csv') >df_case.head(1) sacc_id$ id$ creation_date 0 001A000000hwvV0IAI 5001200000ZnfUgAAJ 2016-06-07 14:38:02 dataframe2: >df_limdata = dd.read_csv('s3://../.../df_limdata.csv') >df_limdata.head(1) sacc_id$ opp_line_id$ oppline_creation_date 0 001A000000hAUn8IAG a0W1200000G0i3UEAR 2015-06-10 First, I did a merge of the 2 dataframes : > case = dd.merge(df_limdata, df_case, left_on='sacc_id$',right_on='sacc_id$') >case

问题 I have a dask dataframe like the following: import dask.dataframe as dd df = dd.read_csv('s3://...') How do I get a list of futures from this dataframe? 回答1: You can use the the .to_delayed method to convert from a dask dataframe to a list of dask.delayed objects L = df.to_delayed() You can then convert these delayed objects into dask futures using the client.compute method. from dask.distributed import Client client = Client() futures = client.compute(L) 来源： https://stackoverflow.com

问题 I have a huge .csv file (above 100 GB) in the form: | Column1 | Column2 | Column3 | Column4 | Column5 | |---------|---------|---------|---------|---------------------| | A | B | 35 | X | 2017-12-19 11:28:34 | | A | C | 22 | Z | 2017-12-19 11:27:24 | | A | B | 678 | Y | 2017-12-19 11:38:36 | | C | A | 93 | X | 2017-12-19 11:44:42 | And want to summarize it by the unique values in Column1 and Column2 with sum(Column3), max(Column5) the value of Column4, where Column5 was at its maximum.

问题 I would like to distribute a larger object (or load from disk) when a worker loads and put it into a global variable (such as calib_data ). Does that work with dask workers? 回答1: Seems like the client method register_worker_callbacks can do what you want in this case. You will still need somewhere to put your variable, since in python there is no truly global scope. That somewhere could be any attribute of an imported module, for example, which, then, any worker would have access to. You

问题 While running Dask 0.16.0 on OSX 10.12.6 I'm unable to connect a local dask-worker to a local dask-scheduler . I simply want to follow the official Dask tutorial. Steps to reproduce: Step 1: run dask-scheduler Step 2: Run dask-worker 10.160.39.103:8786 The problem seems to related to the dask scheduler and not the worker, as I'm not even able to access the port by other means (e.g., nc -zv 10.160.39.103 8786 ). However, the process is clearly still running on the machine: 回答1: My first guess

问题 I've set up a distributed system using dask. When I start the scheduler using the Python API, the dask scheduler doesn't mention starting the dashboard. As expected, I can not reach it on the address I would expect it to be. Since bokeh is installed, I'd expect the dashboard to be started. When I start the scheduler using the command line however, the dashboard starts correctly. Why is it that starting the scheduler through the python api does not start the dashboard? Relevant information:

问题 This is related to how to parallelize many (fuzzy) string comparisons using apply in Pandas? Consider this simple (but funny) example again: import dask.dataframe as dd import dask.multiprocessing import dask.threaded from fuzzywuzzy import fuzz import pandas as pd master= pd.DataFrame({'original':['this is a nice sentence', 'this is another one', 'stackoverflow is nice']}) slave= pd.DataFrame({'name':['hello world', 'congratulations', 'this is a nice sentence ', 'this is another one',

问题 This is a more general question about how to run "embarassingly paralllel" problems with python "schedulers" in a science environment. I have a code that is a Python/Cython/C hybrid (for this example I'm using github.com/tardis-sn/tardis .. but I have more such problems for other codes) that is internally OpenMP parallalized. It provides a single function that takes a parameter dictionary and evaluates to an object within a few hundred seconds running on ~8 cores ( result=fun(paramset,

问题 I have been trying to manage the memory usage of Dask on a single local machine. For some reason, the default Dask Client() and LocalCluster() scheduler always seem to break, however Dask works great without specifying the scheduler and thus the default scheduler works the best for my purposes, however I am finding almost no documentation on this default scheduler let alone how to set a RAM limit on it. All of the information is for their specialized distributed client which does not seem to