dask

I want to submit functions with Dask that have large (gigabyte scale) arguments. What is the best way to do this? I want to run this function many times with different (small) parameters. Example (bad) This uses the concurrent.futures interface. We could use the dask.delayed interface just as easily. x = np.random.random(size=100000000) # 800MB array params = list(range(100)) # 100 small parameters def f(x, param): pass from dask.distributed import Client c = Client() futures = [c.submit(f, x, param) for param in params] But this is slower than I would expect or results in memory errors. OK,

I do the following: import dask.dataframe as dd from dask.distributed import Client client = Client() raw_data_df = dd.read_csv('dataset/nyctaxi/nyctaxi/*.csv', assume_missing=True, parse_dates=['tpep_pickup_datetime', 'tpep_dropoff_datetime']) The dataset is taken out of a presentation Mathew Rocklin has made and was used as a dask dataframe demo. Then I try to write it to parquet using pyarrow raw_data_df.to_parquet(path='dataset/parquet/2015.parquet/') # only pyarrow is installed Trying to read back: raw_data_df = dd.read_parquet(path='dataset/parquet/2015.parquet/') I get the following

Is it possible, to iterate over a dask GroupBy object to get access to the underlying dataframes? I tried: import dask.dataframe as dd import pandas as pd pdf = pd.DataFrame({'A':[1,2,3,4,5], 'B':['1','1','a','a','a']}) ddf = dd.from_pandas(pdf, npartitions = 3) groups = ddf.groupby('B') for name, df in groups: print(name) However, this results in an error: KeyError: 'Column not found: 0' More generally speaking, what kind of interactions does the dask GroupBy object allow, except from the apply method? you could iterate through groups doing this with dask, maybe there is a better way but this

I have files which are made of 10 ensembles and 35 time files. One of these files looks like: >>> xr.open_dataset('ens1/CCSM4_ens1_07ic_19820701-19820731_NPac_Jul.nc') <xarray.Dataset> Dimensions: (ensemble: 1, latitude: 66, longitude: 191, time: 31) Coordinates: * ensemble (ensemble) int32 1 * latitude (latitude) float32 0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0 ... * longitude (longitude) float32 100.0 101.0 102.0 103.0 104.0 105.0 106.0 ... * time (time) datetime64[ns] 1982-07-01 1982-07-02 1982-07-03 ... Data variables: u10m (time, latitude, longitude) float64 -1.471 -0.05933 -1.923 ...

I have a raster time series stored in multiple GeoTIFF files ( *.tif ) that I'd like to convert to a single NetCDF file. The data is uint16 . I could probably use gdal_translate to convert each image to netcdf using: gdal_translate -of netcdf -co FORMAT=NC4 20150520_0164.tif foo.nc and then some scripting with NCO to extract dates from filenames and then concatenate, but I was wondering whether I might do this more effectively in Python using xarray and it's new rasterio backend. I can read a file easily: import glob import xarray as xr f = glob.glob('*.tif') da = xr.open_rasterio(f[0]) da

This is my first venture into parallel processing and I have been looking into Dask but I am having trouble actually coding it. I have had a look at their examples and documentation and I think dask.delayed will work best. I attempted to wrap my functions with the delayed(function_name), or add an @delayed decorator, but I can't seem to get it working properly. I preferred Dask over other methods since it is made in python and for its (supposed) simplicity. I know dask doesn't work on the for loop, but they say it can work inside a loop. My code passes files through a function that contains