Python MemoryError issue on Ubuntu but not on windows when running matpplotlib's specgram command

问题

I would sincerely appreciate your assistance with the following please:

I am trying to run some python code (using Ipython, Python version 2.7.3) on my on a dual booted laptop (- Ubuntu 12.04 LTS, 64 bit and MS Windows 7 64 bit). Running the code in windows works fine, but for some reason gives me memory errors in Ubuntu. I'm not sure if this is a python issue or an OS problem. I would stick with running the code on MS windows, but that has it's own issues which I'm struggling with, including importing libraries etc. which I don't have when running the code in Ubuntu.

The code is reasonably straight forward. For reference, I have a large binary file, which represents 16 bit signed numbers from 3 channels in a lab. instrument. Data was sampled at 10 MS/s. The file is ~ 1.G Gbytes which represents roughly 25 seconds of data. In the code below I read in the data into "data" and would now like to plot the spectrogram using Matplotlib's specgram function. This part works fine in Windows but not Ubuntu.

some code...

import os
import sys
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.mlab as mlab
file = "data_stream.srm"
with open(file,'rb') as fd:
fd.seek(0)
data = np.fromfile(fd,dtype=[('ch0','<i2'),('ch1','<i2'),('ch2','<i2')])

The data contents (raw signal values) look like:

array([(-1, 5, -3), (-4, 3, -4), (-6, 0, -3), ..., (-1, -2, 0),
   (-2, -1, -2), (-2, -2, -1)], 
  dtype=[('ch0', '<i2'), ('ch1', '<i2'), ('ch2', '<i2')])

Possibly not the best way to do it but it works fine up to here (on both Ubuntu and Windows). Size of data (using data.shape) gives (261144000,). Would have expected to see that the shape was (261144000,3) as this large number is the number of rows (and hence number of values) for each channel of which there are 3. Typing for example:

len(data'ch0'])

which returns, 261144000

Next issue is the spectrogram using matplotlib's specgram function

   specgram(x, NFFT=256, Fs=2, Fc=0, detrend=mlab.detrend_none,
    window=mlab.window_hanning, noverlap=128,
    cmap=None, xextent=None, pad_to=None, sides='default',
    scale_by_freq=None, **kwargs)

some code ...

ADCR = 12        # ADC Resolution in bits
    VOLT = 5.0       # Voltage range
    SF = VOLT/(2**ADCR)  # scaling factor to convert raw ADC values to volts  

    w_length= 512 
    nFFT=2 * w_length
    n_overlap=np.fix(w_length/2)
    p_to = 4 *w_length
    cmap=plt.cm.seismic
    Fs = 10E6 # Sampling Frequency
    fig=plt.figure()
    ax=fig.add_subplot(111)
    Pxx, freqs, bins,im =    


plt.specgram((data['ch1'])*SF,NFFT=nFFT,Fs=Fs,noverlap=n_overlap,pad_to=p_to,cmap=cmap)

Running this last command works in windows but not Ubuntu. I get the following messages:

Memory Error... "can't post an image as I need 10 reputation to post"

MemoryError                               Traceback (most recent call last)
/media/IomegaHDD/<ipython-input-28-5144fe0c6918> in <module>()
----> 1 Pxx, freqs, bins, im = plt.specgram((data['ch1'])*SF,NFFT=nFFT,Fs=Fs,detrend=mlab.detrend_linear,noverlap=n_overlap,pad_to=p_to,scale_by_freq=True,cmap=cmap)

/usr/lib/pymodules/python2.7/matplotlib/pyplot.pyc in specgram(x, NFFT, Fs, Fc, detrend, window, noverlap, cmap, xextent, pad_to, sides, scale_by_freq, hold, **kwargs)
   2609         ax.hold(hold)
   2610     try:
-> 2611         ret = ax.specgram(x, NFFT, Fs, Fc, detrend, window, noverlap, cmap, xextent, pad_to, sides, scale_by_freq, **kwargs)
   2612         draw_if_interactive()
   2613     finally:

/usr/lib/pymodules/python2.7/matplotlib/axes.pyc in specgram(self, x, NFFT, Fs, Fc, detrend, window, noverlap, cmap, xextent, pad_to, sides, scale_by_freq, **kwargs)
   8146 
   8147         Pxx, freqs, bins = mlab.specgram(x, NFFT, Fs, detrend,
-> 8148              window, noverlap, pad_to, sides, scale_by_freq)
   8149 
   8150         Z = 10. * np.log10(Pxx)

/usr/lib/pymodules/python2.7/matplotlib/mlab.pyc in specgram(x, NFFT, Fs, detrend, window, noverlap, pad_to, sides, scale_by_freq)
    458 
    459     Pxx, freqs, t = _spectral_helper(x, x, NFFT, Fs, detrend, window,
--> 460         noverlap, pad_to, sides, scale_by_freq)
    461     Pxx = Pxx.real #Needed since helper implements generically
    462 

/usr/lib/pymodules/python2.7/matplotlib/mlab.pyc in _spectral_helper(x, y, NFFT, Fs, detrend, window, noverlap, pad_to, sides, scale_by_freq)
    256     ind = np.arange(0, len(x) - NFFT + 1, step)
    257     n = len(ind)
--> 258     Pxy = np.zeros((numFreqs, n), np.complex_)
    259 
    260     # do the ffts of the slices


MemoryError:

As I said perhaps it's an Ubuntu issue I don't know as I don't get this problem running the code in Windows. This however has it's own issues with other subsequent parts of the code.

Your assistance would be appreciated.

Thank you.

Regards,

Will.

回答1:

Are you sure you're running the 32 bit Ubuntu distribution? You must also have a 64 bit version of python running too.

If this is the problem, it's down to the fact that 32 bit operating systems (and programs) can't address the large memory chunks that 64 bit systems can.

To tell if you have 64 bit Linux, run (in Ubuntu)