dask-distributed

问题 I am using Dask behind a Django server and the basic setup I have is summarised here: https://github.com/MoonVision/django-dask-demo/ where the Dask client can be found here: https://github.com/MoonVision/django-dask-demo/blob/master/demo/daskmanager/daskmanager.py I want to be able to separate the saving of a task from the server that submitted it for robustness and scalability. I also would like more detailed information as to the processing status of the task, right now the future status

问题 I am aware of this question. But check the code(minimal-working example) below: import dask.dataframe as dd import pandas as pd # intialise data of lists. data = {'Name': ['Tom', 'nick', 'krish', 'jack'], 'Age': [20, 21, 19, 18]} # Create DataFrame df = pd.DataFrame(data) dask_df = dd.from_pandas(df, npartitions=1) categoric_df = dask_df.select_dtypes(include="category") When I try to print the categoric_df I get the following error: ValueError: No objects to concatenate And when I check the

问题 I am aware of this question. But check the code(minimal-working example) below: import dask.dataframe as dd import pandas as pd # intialise data of lists. data = {'Name': ['Tom', 'nick', 'krish', 'jack'], 'Age': [20, 21, 19, 18]} # Create DataFrame df = pd.DataFrame(data) dask_df = dd.from_pandas(df, npartitions=1) categoric_df = dask_df.select_dtypes(include="category") When I try to print the categoric_df I get the following error: ValueError: No objects to concatenate And when I check the

问题 I have a docker-compose.yml : version: '2' services: scheduler: build: context: . dockerfile: Dockerfile hostname: dask-scheduler ports: - "8786:8786" - "8787:8787" command: dask-scheduler worker: build: context: . dockerfile: Dockerfile hostname: dask-worker ports: - "8789:8789" command: dask-worker scheduler:8786 and the Dockerfile : FROM continuumio/miniconda3 RUN apt-get update && apt-get install -y build-essential freetds-dev RUN mkdir project COPY requirements.txt /project/requirements

问题 While running Dask 0.16.0 on OSX 10.12.6 I'm unable to connect a local dask-worker to a local dask-scheduler . I simply want to follow the official Dask tutorial. Steps to reproduce: Step 1: run dask-scheduler Step 2: Run dask-worker 10.160.39.103:8786 The problem seems to related to the dask scheduler and not the worker, as I'm not even able to access the port by other means (e.g., nc -zv 10.160.39.103 8786 ). However, the process is clearly still running on the machine: 回答1: My first guess

问题 I've set up a distributed system using dask. When I start the scheduler using the Python API, the dask scheduler doesn't mention starting the dashboard. As expected, I can not reach it on the address I would expect it to be. Since bokeh is installed, I'd expect the dashboard to be started. When I start the scheduler using the command line however, the dashboard starts correctly. Why is it that starting the scheduler through the python api does not start the dashboard? Relevant information:

问题 This is a more general question about how to run "embarassingly paralllel" problems with python "schedulers" in a science environment. I have a code that is a Python/Cython/C hybrid (for this example I'm using github.com/tardis-sn/tardis .. but I have more such problems for other codes) that is internally OpenMP parallalized. It provides a single function that takes a parameter dictionary and evaluates to an object within a few hundred seconds running on ~8 cores ( result=fun(paramset,

问题 I have been trying to manage the memory usage of Dask on a single local machine. For some reason, the default Dask Client() and LocalCluster() scheduler always seem to break, however Dask works great without specifying the scheduler and thus the default scheduler works the best for my purposes, however I am finding almost no documentation on this default scheduler let alone how to set a RAM limit on it. All of the information is for their specialized distributed client which does not seem to

问题 I am experimenting with Dask and want to ship a lookup pandas.DataFrame to all worker nodes. Unfortunately, it fails with: TypeError: ("'Future' object is not subscriptable", 'occurred at index 0') When instead of lookup['baz'].iloc[2] using lookup.result()['foo'].iloc[2] , it works fine but: for larger instances of the input dataframe, it seems to be stuck at from_pandas again and again. Also, it seems strange that the future needs to be blocked manually (over and over again for each row in

问题 I have some code that uses Numba cuda.jit in order for me to run on the gpu, and I would like to layer dask on top of it if possible. Example Code #!/usr/bin/env python3 # -*- coding: utf-8 -*- from numba import cuda, njit import numpy as np from dask.distributed import Client, LocalCluster @cuda.jit() def addingNumbersCUDA (big_array, big_array2, save_array): i = cuda.grid(1) if i < big_array.shape[0]: for j in range (big_array.shape[1]): save_array[i][j] = big_array[i][j] * big_array2[i][j]