dask

I am trying to read parquet files using the dask read_parquet method and the filters kwarg. however it sometimes doesn't filter according to the given condition. Example: creating and saving data frame with a dates column import pandas as pd import numpy as np import dask.dataframe as dd nums = range(1,6) dates = pd.date_range('2018-07-01', periods=5, freq='1d') df = pd.DataFrame({'dates':dates, 'nums': nums}) ddf = dd.from_pandas(df, npartitions=3).to_parquet('test_par', engine = 'fastparquet') when i read and filter on the dates column from the 'test_par' folder it doesn't seem to work

Often, when working with a large dask.DataFrame , it would be useful to grab only a few rows on which to test all subsequent operations. Currently, according to Slicing a Dask Dataframe , this is unsupported. I was hoping to then use head to achieve the same (since that command is supported), but that returns a regular pandas DataFrame. I also tried df[:1000] , which executes, but generates an output different from that you'd expect from Pandas. Is there any way to grab the first 1000 rows from a dask.DataFrame ? If your dataframe has a sensibly partitioned index then I recommend using .loc

I am looking for a way to remove rows from a dataframe that contain low frequency items. I adapted the following snippet from this post: import pandas as pd import numpy as np df = pd.DataFrame(np.random.randint(0, high=9, size=(100,2)), columns = ['A', 'B']) threshold = 10 # Anything that occurs less than this will be removed. value_counts = df.stack().value_counts() # Entire DataFrame to_remove = value_counts[value_counts <= threshold].index df.replace(to_remove, np.nan, inplace=True) The problem is, that this code does not scale, it seems. The line to_remove = value_counts[value_counts <=

If I have a Pandas dataframe, and a column that is a datetime type, I can get the year as follows: df['year'] = df['date'].dt.year With a dask dataframe, that does not work. If I compute first, like this: df['year'] = df['date'].compute().dt.year I get ValueError: Not all divisions are known, can't align partitions. Please use set_index or set_partition to set the index. But if I do: df['date'].head().dt.year it works fine! So how do I get the year (or week) of a datetime series in a dask dataframe? The .dt datetime namespace is present on Dask series objects. Here is a self-contained of its

I have a list of the URLs, and I'd love to read them to the dask data frame at once, but it looks like read_csv can't use an asterisk for http . Is there any way to achieve that? Here is an example: link = 'http://web.mta.info/developers/' data = [ 'data/nyct/turnstile/turnstile_170128.txt', 'data/nyct/turnstile/turnstile_170121.txt', 'data/nyct/turnstile/turnstile_170114.txt', 'data/nyct/turnstile/turnstile_170107.txt' ] and what I want is df = dd.read_csv('XXXX*X') Try using dask.delayed to turn each of your urls into a lazy pandas dataframe and then use dask.dataframe.from_delayed to turn

I want to find an alternative of pandas.dataframe.sort_value function in dask. I came through set_index , but it would sort on a single column. How can I sort multiple columns of Dask data frame? So far Dask does not seem to support sorting by multiple columns. However, making a new column that concatenates the values of the sorted columns may be a usable work-around. d['new_column'] = d.apply(lambda r: str([r.col1,r.col2]), axis=1) d = d.set_index('new_column') d = d.map_partitions(lambda x: x.sort_index()) Edit: The above works if you want to sort by two strings. I recommend creating integer

I am trying to load a dataset with dask but when it is time to compute my dataset I keep getting problems like this: WARNING - Worker exceeded 95% memory budget. Restarting. I am just working on my local machine, initiating dask as follows: if __name__ == '__main__': libmarket.config.client = Client() # use dask.distributed by default Now in my error messages I keep seeing a reference to a 'memory_limit=' keyword parameter. However I've searched the dask documentation thoroughly and I can't figure out how to increase the bloody worker memory-limit in a single-machine configuration. I have