Python pandas error while removing extra white space

问题

I am trying to clean a column in data frame of extra white space using command. The data frame has close to 8 million records

datt2.My_variable=datt2.My_variable.str.replace('\s+', ' ')

I end up getting below error

MemoryError                               Traceback (most recent call last)
<ipython-input-10-158a51cfaa3d> in <module>()
----> 1 datt2.My_variable=datt2.My_variable.str.replace('\s+', ' ')

c:\python27\lib\site-packages\pandas\core\strings.pyc in replace(self, pat, repl, n, case, flags)
   1504     def replace(self, pat, repl, n=-1, case=True, flags=0):
   1505         result = str_replace(self._data, pat, repl, n=n, case=case,
-> 1506                              flags=flags)
   1507         return self._wrap_result(result)
   1508 

c:\python27\lib\site-packages\pandas\core\strings.pyc in str_replace(arr, pat, repl, n, case, flags)
    334         f = lambda x: x.replace(pat, repl, n)
    335 
--> 336     return _na_map(f, arr)
    337 
    338 

c:\python27\lib\site-packages\pandas\core\strings.pyc in _na_map(f, arr, na_result, dtype)
    152 def _na_map(f, arr, na_result=np.nan, dtype=object):
    153     # should really _check_ for NA
--> 154     return _map(f, arr, na_mask=True, na_value=na_result, dtype=dtype)
    155 
    156 

c:\python27\lib\site-packages\pandas\core\strings.pyc in _map(f, arr, na_mask, na_value, dtype)
    167         try:
    168             convert = not all(mask)
--> 169             result = lib.map_infer_mask(arr, f, mask.view(np.uint8), convert)
    170         except (TypeError, AttributeError):
    171 

pandas\src\inference.pyx in pandas.lib.map_infer_mask (pandas\lib.c:65837)()

pandas\src\inference.pyx in pandas.lib.maybe_convert_objects (pandas\lib.c:56806)()

MemoryError:

回答1:

Question: I am trying to clean a column in data frame of extra white space ...
datt2.My_variable=datt2.My_variable.str.replace('\s+', ' ')

Please comment, do I understand your expression correctly?

 pandas       Column           Column              DataSeries
 DataFrame     Name           DataSeries             Methode
|-^-|       |----^-----|   |-------^-------|  |----------^----------|
datt2       .My_variable = datt2.My_variable  .str.replace('\s+', ' ')

---

I'm pretty sure using re.sub is the same as use pandas.str.replace(...), but without copy the whole column Data.

From the pandas doc:
Series.str.replace(pat, repl, n=-1, case=True, flags=0)
Replace occurrences of pattern/regex in the Series/Index with some other string.
Equivalent to str.replace() or re.sub().

---

Try pure python, for instance:

    import re
    for idx in df.index:
        df.loc[idx, 'My_variable'] = re.sub('\s\s+', ' ', df.loc[idx, 'My_variable'])  

Note: Consider to use '\s\s+' instead of '\s+'.
Using '\s+' will replace ONE BLANK with ONE BLANK, which is useless.

Tested with Python:3.4.2 - pandas:0.19.2
Come back and Flag your Question as answered if this is working for you or comment why not.

来源：https://stackoverflow.com/questions/42854573/python-pandas-error-while-removing-extra-white-space