Adding datetime field to recarray

Question

I am trying to append a date-time field (datetime64) to an existing recarray - without much success. I can create the datetime field, but when I attempt to append it to the

Answer 1

The traceback is:

Traceback (most recent call last):
  File "stack26739733.py", line 30, in <module>
    datarray = rf.append_fields(datarray, 'DateTime', data=mydatetime, usemask=False, dtypes=mydatetime.dtype)
  File "/usr/local/lib/python2.7/site-packages/numpy/lib/recfunctions.py", line 641, in append_fields
    dtype=base.dtype.descr + data.dtype.descr)
  File "/usr/local/lib/python2.7/site-packages/numpy/ma/extras.py", line 163, in masked_all
    mask=np.ones(shape, make_mask_descr(dtype)))
  File "/usr/local/lib/python2.7/site-packages/numpy/ma/core.py", line 2644, in __new__
    _data = ndarray.view(_data, cls)
  File "/usr/local/lib/python2.7/site-packages/numpy/ma/core.py", line 2800, in __array_finalize__
    self._fill_value = _check_fill_value(None, self.dtype)
  File "/usr/local/lib/python2.7/site-packages/numpy/ma/core.py", line 402, in _check_fill_value
    dtype=ndtype,)
ValueError: Error parsing datetime string "?" at position 0

So this append function constructs a masked array (ma), and checks the 'fill_value' for the appended 'dtype'. Apparently _check_fill_value doesn't understand the datetime dtype. Looks like an incompatibility between masked array and datetime. A numpy bug report might be in order.

---

Here's a simple, do-it-yourself append:

dt1 = np.dtype(datarray.dtype.descr + mydatetime.dtype.descr)
newarray = np.empty(datarray.shape, dtype=dt1)
for n in datarray.dtype.names:
    newarray[n] = datarray[n]
newarray['f3'] = mydatetime

I construct an empty array with a union dtype. Then I copy the data from both datarray and mydatetime field by field. Since the number of fields is normally quite small compared to the shape, this copy is quite fast. I'm pretty sure the rf function does the same.

'f3' is the default name of the added field. You can change that when creating dt1.

The result is:

array([(0, 0, 0, datetime.datetime(1990, 1, 1, 0, 0)),
       (1, 1, 1, datetime.datetime(1990, 1, 1, 0, 0, 1, 1000)),
       (2, 2, 2, datetime.datetime(1990, 1, 1, 0, 0, 2, 2000)),
       ...
       (9, 9, 9, datetime.datetime(1990, 1, 1, 0, 0, 9, 9000))], 
      dtype=[('a', '<i4'), ('b', '<i4'), ('c', '<i4'), ('f3', '<M8[ms]')])

Turning this newarray into a masked array produces the same _check_fill_value error.

np.ma.masked_array(newarray)