dask-distributed

问题 What exactly I want to achieve in terms of input output is a cross - join Input Example df = pd.DataFrame(columns = ['A', 'val'], data = [['a1', 23],['a2', 29], ['a3', 39]]) print(df) A val 0 a1 23 1 a2 29 2 a3 39 Output Example: df['key'] = 1 df.merge(df, how = "outer", on ="key") A_x val_x key A_y val_y 0 a1 23 1 a1 23 1 a1 23 1 a2 29 2 a1 23 1 a3 39 3 a2 29 1 a1 23 4 a2 29 1 a2 29 5 a2 29 1 a3 39 6 a3 39 1 a1 23 7 a3 39 1 a2 29 8 a3 39 1 a3 39 How I achieve this for a large dataset with

问题 What exactly I want to achieve in terms of input output is a cross - join Input Example df = pd.DataFrame(columns = ['A', 'val'], data = [['a1', 23],['a2', 29], ['a3', 39]]) print(df) A val 0 a1 23 1 a2 29 2 a3 39 Output Example: df['key'] = 1 df.merge(df, how = "outer", on ="key") A_x val_x key A_y val_y 0 a1 23 1 a1 23 1 a1 23 1 a2 29 2 a1 23 1 a3 39 3 a2 29 1 a1 23 4 a2 29 1 a2 29 5 a2 29 1 a3 39 6 a3 39 1 a1 23 7 a3 39 1 a2 29 8 a3 39 1 a3 39 How I achieve this for a large dataset with

问题 Is it possible to run dask from a python script? In interactive session I can just write from dask.distributed import Client client = Client() as described in all tutorials. If I write these lines however in a script.py file and execute it python script.py , it immediately crashes. I found another option I found, is to use MPI: # script.py from dask_mpi import initialize initialize() from dask.distributed import Client client = Client() # Connect this local process to remote workers And then

问题 I am trying to pass more data into my GPU than I have VRAM, which results in the following error. CudaAPIError: Call to cuMemAlloc results in CUDA_ERROR_OUT_OF_MEMORY I created this code to recreate the problem: from numba import cuda import numpy as np @cuda.jit() def addingNumbers (big_array, big_array2, save_array): i = cuda.grid(1) if i < big_array.shape[0]: for j in range (big_array.shape[1]): save_array[i][j] = big_array[i][j] * big_array2[i][j] big_array = np.random.random_sample(

问题 I am trying to pass more data into my GPU than I have VRAM, which results in the following error. CudaAPIError: Call to cuMemAlloc results in CUDA_ERROR_OUT_OF_MEMORY I created this code to recreate the problem: from numba import cuda import numpy as np @cuda.jit() def addingNumbers (big_array, big_array2, save_array): i = cuda.grid(1) if i < big_array.shape[0]: for j in range (big_array.shape[1]): save_array[i][j] = big_array[i][j] * big_array2[i][j] big_array = np.random.random_sample(

问题 I am trying to pass more data into my GPU than I have VRAM, which results in the following error. CudaAPIError: Call to cuMemAlloc results in CUDA_ERROR_OUT_OF_MEMORY I created this code to recreate the problem: from numba import cuda import numpy as np @cuda.jit() def addingNumbers (big_array, big_array2, save_array): i = cuda.grid(1) if i < big_array.shape[0]: for j in range (big_array.shape[1]): save_array[i][j] = big_array[i][j] * big_array2[i][j] big_array = np.random.random_sample(

问题 From the docs, there is a class called Cluster , but other than for the LocalCluster , I can not find any docs about how to setup a Cluster which accepts workers from different machines (as this is described for LocalCluster here. Are there some recommendations out there? Related question: 1 (only reference on the docs as above) 回答1: The answer is SSHCluster It is based on asyncssh and can have multiple workers per machine in the cluster. Make sure to have pre requirements installed. 来源：

问题 I'm having trouble changing the temporary directory in Dask. When I change the temporary-directory in dask.yaml for some reason Dask is still writing out in /tmp (which is full). I now want to try and debug this, but when I use client.get_worker_logs() I only get INFO output. I start my cluster with from dask.distributed import LocalCluster, Client cluster = LocalCluster(n_workers=1, threads_per_worker=4, memory_limit='10gb') client = Client(cluster) I already tried adding distributed.worker:

问题 Context I'm using dask.distributed to parallelise computations across machines. I therefore have dask-workers running on the different machines which connect to a dask-scheduler, to which I can then submit my custom graphs to together with the required keys. Due to network mount restrictions, my input data (and output storage) is only available to a subset of the machines ('i/o-hosts'). I tried to deal with this in two ways: all tasks involved in i/o operations are restricted to i/o-hosts

问题 I am new to dask and I found so nice to have a module that makes it easy to get parallelization. I am working on a project where I was able to parallelize in a single machine a loop as you can see here . However, I would like to move over to dask.distributed . I applied the following changes to the class above: diff --git a/mlchem/fingerprints/gaussian.py b/mlchem/fingerprints/gaussian.py index ce6a72b..89f8638 100644 --- a/mlchem/fingerprints/gaussian.py +++ b/mlchem/fingerprints/gaussian.py