numexpr

Pytables NumExpr ValueError: too many inputs when querying with a lot of conditions

由 两盒软妹~` 提交于 2020-05-17 03:40:06
问题 I encounter this error when querying a pytables table with where method passing a string with 50 conditions. These conditions are basically a translation of a SQL IN clause i.e. I want to get records where a particular field is equal to a certain list of values. C:\Python2764\lib\site-packages\numexpr\necompiler.pyc in evaluate(ex, local_dict, global_dict, out, order, casting, **kwargs) 744 kwargs = {'out': out, 'order': order, 'casting': casting, 745 'ex_uses_vml': ex_uses_vml} --> 746

Numexpr not found in pandas

由 南楼画角 提交于 2020-01-06 02:41:13
问题 I install numexpr package via pip on my Windows 7 machine: pip list | grep numexpr numexpr (2.4.6) but when I open ipython and trying to use df.query it shows an error: ImportError: 'numexpr' not found. Cannot use engine='numexpr' for query/eval if 'numexpr' is not installed Than I checked whether it installed or not with pd.show_versions: In [97]: pd.show_versions() INSTALLED VERSIONS ------------------ commit: None python: 3.4.3.final.0 python-bits: 32 OS: Windows OS-release: 7 machine:

What is the most memory efficient way to combine read_sorted and Expr in pytables?

由 白昼怎懂夜的黑 提交于 2020-01-05 11:03:09
问题 I am looking for the most memory efficient way to combine reading a Pytables table (columns: x,y,z) in a sorted order(z column has a CSI) and evaluating an expression like x+a*y+b*z where a and b are constant. Up until now my only solution was to copy the entire table with the "sortyby=z" flag and then evaluating the expression piece-wise on the table. Note: I want to keep the result x+a*y+b*z in memory to do some reduction operations on it which are not available directly in Pytables and

Numexpr doesn't recognize float type (sparse matrix)

由 余生颓废 提交于 2019-12-24 13:26:10
问题 I would like to evaluate the performance of numexpr module in python (2.7). For that purpose, I created a random sparse matrix of size (10^5, 10^5). However, the script below throws an error at the expression evaluation step already, saying that it doesn't recognize the object type. What am I doing wrong? Code: import timeit import scipy.sparse as sps import numpy as np import numexpr as ne test_matrix = sps.rand(1e4, 1e4, density=0.01, format='coo', dtype = np.float32) ne.evaluate('sum(test

Numexpr: How to use “local_dict” and “global_dict”?

由 笑着哭i 提交于 2019-12-23 13:03:14
问题 I have been experimenting and trying to learn the Numexpr package. Examples on how to use it are sparse at best. Can someone please give me a quick example on how to use the "local_dict" and "global_dict" arguments? 回答1: The following examples may clarify it a little. First set up the scene like this: import numpy as np import numexpr as ne a = np.arange(10) b = np.arange(10, 20) c = np.arange(20, 30) No dict >>> def f0(a, b) : ... a, b = 2 * a, 3 * b ... return ne.evaluate('2*a + 3*b + c') .

Parallel in-place sort for numpy arrays

由 混江龙づ霸主 提交于 2019-12-04 18:59:01
问题 I often need to sort large numpy arrays (few billion elements), which became a bottleneck of my code. I am looking for a way to parallelize it. Are there any parallel implementations for the ndarray.sort() function? Numexpr module provides parallel implementation for most math operations on numpy arrays, but lacks sorting capabilities. Maybe, it is possible to make a simple wrapper around a C++ implementation of parallel sorting, and use it through Cython? 回答1: I ended up wrapping GCC

Parallel in-place sort for numpy arrays

由 微笑、不失礼 提交于 2019-12-03 12:31:37
I often need to sort large numpy arrays (few billion elements), which became a bottleneck of my code. I am looking for a way to parallelize it. Are there any parallel implementations for the ndarray.sort() function? Numexpr module provides parallel implementation for most math operations on numpy arrays, but lacks sorting capabilities. Maybe, it is possible to make a simple wrapper around a C++ implementation of parallel sorting, and use it through Cython? I ended up wrapping GCC parallel sort. Here is the code: parallelSort.pyx # cython: wraparound = False # cython: boundscheck = False import

python - way to do fast matrix multiplication and reduction while working in memmaps and CPU

由 好久不见. 提交于 2019-12-02 05:39:10
问题 Hi i have problem doing fast matrix multiplication, addition,function_overwrite and summation with axis reduction and working in numpy.memmaps over CPU without RAM (I think). Only when using numexpr it is possible for me to avoid creating array from dot. For example: a=np.require(np.memmap('a.npy',mode='w+',order='C',dtype=np.float64,shape=(10,1)),requirements=['O']) b=np.memmap('b.npy',mode='w+',order='C',dtype=np.float64,shape=(1,5)) c=np.memmap('c.npy',mode='w+',order='C',dtype=np.float64

python - way to do fast matrix multiplication and reduction while working in memmaps and CPU

由 ◇◆丶佛笑我妖孽 提交于 2019-12-02 00:01:29
Hi i have problem doing fast matrix multiplication, addition,function_overwrite and summation with axis reduction and working in numpy.memmaps over CPU without RAM (I think). Only when using numexpr it is possible for me to avoid creating array from dot. For example: a=np.require(np.memmap('a.npy',mode='w+',order='C',dtype=np.float64,shape=(10,1)),requirements=['O']) b=np.memmap('b.npy',mode='w+',order='C',dtype=np.float64,shape=(1,5)) c=np.memmap('c.npy',mode='w+',order='C',dtype=np.float64,shape=(1,5)) #func -> some method, like i.e. sin() #in numexpr it will be simple ne.evaluate('sum(func

numba guvectorize target='parallel' slower than target='cpu'

由 杀马特。学长 韩版系。学妹 提交于 2019-11-27 16:56:44
问题 I've been attempting to optimize a piece of python code that involves large multi-dimensional array calculations. I am getting counterintuitive results with numba. I am running on an MBP, mid 2015, 2.5 GHz i7 quadcore, OS 10.10.5, python 2.7.11. Consider the following: import numpy as np from numba import jit, vectorize, guvectorize import numexpr as ne import timeit def add_two_2ds_naive(A,B,res): for i in range(A.shape[0]): for j in range(B.shape[1]): res[i,j] = A[i,j]+B[i,j] @jit def add
订阅 numexpr