问题
I'm new to dask so could you help me please? I have a csv-file like this:
id,popularity,hashtag,seen
0,100,#footbal,0
1,200,#2017,0
2,300,#1,0
and somehow i managed to get a dask dataframe hashtags_to_update:
id seen
0 118
2 136
I'd like to merge a data from hashtags_to_update with data from csv-file to get:
id,popularity,hashtag,seen
0,100,#footbal,118
1,200,#2017,0
2,300,#1,136
For now I'm doing the following
hashtags_df = dd.read_csv('path/to/csv/file').set_index('id')
hashtags_df["seen"] = hashtags_df["seen"].add(hashtags_to_update["seen"], fill_value=0).astype('int64')
hashtags_df.compute().to_csv('output.csv', sep=',')
But as far as I know there are some problems when the data contains strings which are casted as python's objects, so there will be no parallelism because of GIL.
Is there anything you could advice me to do? Thank you in advance.
回答1:
you can use multiprocessing (thus avoiding the GIL).
there are several ways:
setup a client (by default it will ensure multiprocessing):
from dask.distributed import Client
client = Client()
or
import dask.multiprocessing
dask.config.set(scheduler='processes') # overwrite default with multiprocessing scheduler
according to the documentation the Former is recommended.
more info:
Client
dask.config.set
来源：https://stackoverflow.com/questions/51262480/update-of-dasks-dataframe