ValueError: Data must be positive (boxcox scipy)

Question

I\'m trying to transform my dataset to a normal distribution.

0      8.298511e-03
1      3.055319e-01
2      6.938647e-02
3      2.904091e-02
4      7.422441e-0         


        

Answer 1

Your data contains the value 0 (at index 134). When boxcox says the data must be positive, it means strictly positive.

What is the meaning of your data? Does 0 make sense? Is that 0 actually a very small number that was rounded down to 0?

You could simply discard that 0. Alternatively, you could do something like the following. (This amounts to temporarily discarding the 0, and then using -1/λ for the transformed value of 0, where λ is the Box-Cox transformation parameter.)

First, create some data that contains one 0 (all other values are positive):

In [13]: np.random.seed(8675309)

In [14]: data = np.random.gamma(1, 1, size=405)

In [15]: data[100] = 0

(In your code, you would replace that with, say, data = df.values.)

Copy the strictly positive data to posdata:

In [16]: posdata = data[data > 0]

Find the optimal Box-Cox transformation, and verify that λ is positive. This work-around doesn't work if λ ≤ 0.

In [17]: bcdata, lam = boxcox(posdata)

In [18]: lam
Out[18]: 0.244049919975582

Make a new array to hold that result, along with the limiting value of the transform of 0 (which is -1/λ):

In [19]: x = np.empty_like(data)

In [20]: x[data > 0] = bcdata

In [21]: x[data == 0] = -1/lam

The following plot shows the histograms of data and x.