inequality comparison of numpy array with nan to a scalar

Question

I am trying to set members of an array that are below a threshold to nan. This is part of a QA/QC process and the incoming data may already have slots that are nan.

Answer 1

np.less() has a where argument that controls where the operation will be applied. So you could do:

x[np.less(x, -1000., where=~np.isnan(x))] = np.nan

Answer 2

Any comparison (other than !=) of a NaN to a non-NaN value will always return False:

>>> x < -1000
array([False, False, False,  True, False, False], dtype=bool)

So you can simply ignore the fact that there are NaNs already in your array and do:

>>> x[x < -1000] = np.nan
>>> x
array([ nan,   1.,   2.,  nan,  nan,   5.])

EDIT I don't see any warning when I ran the above, but if you really need to stay away from the NaNs, you can do something like:

mask = ~np.isnan(x)
mask[mask] &= x[mask] < -1000
x[mask] = np.nan

Answer 3

One option is to disable the relevant warnings with numpy.errstate:

with numpy.errstate(invalid='ignore'):
    ...

To turn off the relevant warnings globally, use numpy.seterr.

Answer 4

A little bit late, but this is how I would do:

x = np.array([np.nan,1.,2.,-3000.,np.nan,5.]) 

igood=np.where(~np.isnan(x))[0]
x[igood[x[igood]<-1000.]]=np.nan

Answer 5

I personally ignore the warnings using the np.errstate context manager in the answer already given, as the code clarity is worth the extra time, but here is an alternative.

# given
x = np.array([np.nan, 1., 2., -3000., np.nan, 5.])

# apply NaNs as desired
mask = np.zeros(x.shape, dtype=bool)
np.less(x, -1000, out=mask, where=~np.isnan(x))
x[mask] = np.nan

# expected output and comparison
y = np.array([np.nan, 1., 2., np.nan, np.nan, 5.])
assert np.allclose(x, y, rtol=0., atol=1e-14, equal_nan=True)

The numpy less ufunc takes the optional argument where, and only evaluates it where true, unlike the np.where function which evaluates both options and then picks the relevant one. You then set the desired output when it's not true by using the out argument.