Python: Converting string to timestamp with microseconds

Question

I would like to convert string date format to timestamp with microseconds I try the following but not giving expected result:

"""input string date -> 2014-08-01 04:41:52,117
expected result -> 1410748201.117"""

import time
import datetime

myDate = "2014-08-01 04:41:52,117"
timestamp = time.mktime(datetime.datetime.strptime(myDate, "%Y-%m-%d %H:%M:%S,%f").timetuple())

print timestamp
> 1410748201.0

Where did the milliseconds go?

Answer 1

There is no slot for the microseconds component in a time tuple:

>>> import time
>>> import datetime
>>> myDate = "2014-08-01 04:41:52,117"
>>> datetime.datetime.strptime(myDate, "%Y-%m-%d %H:%M:%S,%f").timetuple()
time.struct_time(tm_year=2014, tm_mon=8, tm_mday=1, tm_hour=4, tm_min=41, tm_sec=52, tm_wday=4, tm_yday=213, tm_isdst=-1)

You'll have to add those manually:

>>> dt = datetime.datetime.strptime(myDate, "%Y-%m-%d %H:%M:%S,%f")
>>> time.mktime(dt.timetuple()) + (dt.microsecond / 1000000.0)
1406864512.117

The other method you could follow is to produce a timedelta() object relative to the epoch, then get the timestamp with the timedelta.total_seconds() method:

epoch = datetime.datetime.fromtimestamp(0)
(dt - epoch).total_seconds()

The use of a local time epoch is quite deliberate since you have a naive (not timezone-aware) datetime value. This method can be inaccurate based on the history of your local timezone however, see J.F. Sebastian's comment. You'd have to convert the naive datetime value to a timezone-aware datetime value first using your local timezone before subtracting a timezone-aware epoch.

As such, it is easier to stick to the timetuple() + microseconds approach.

Demo:

>>> dt = datetime.datetime.strptime(myDate, "%Y-%m-%d %H:%M:%S,%f")
>>> epoch = datetime.datetime.fromtimestamp(0)
>>> (dt - epoch).total_seconds()
1406864512.117

Answer 2

In Python 3.4 and later you can use

timestamp = datetime.datetime.strptime(myDate, "%Y-%m-%d %H:%M:%S,%f").timestamp()

This doesn't require importing the time module. It also uses less steps so it should be faster. For older versions of python the other provided answers are probably your best option.

However, the resulting timestamp will interpret myDate in local time, rather than UTC, which may cause issues if myDate was given in UTC

Answer 3

Where did the milliseconds go?

It is the easy part. .timetuple() call drops them. You could add them back using .microsecond attribute. The datetime.timestamp() method from the standard library works that way for naive datetime objects:

def timestamp(self):
    "Return POSIX timestamp as float"
    if self._tzinfo is None:
        return _time.mktime((self.year, self.month, self.day,
                             self.hour, self.minute, self.second,
                             -1, -1, -1)) + self.microsecond / 1e6
    else:
        return (self - _EPOCH).total_seconds()

It is enough if possible ~1 hour errors could be ignored in your case. I assume that you want microseconds and therefore you can't ignore ~1 hour time errors silently.

To convert the local time given as a string to the POSIX timestamp correctly is a complex task in general. You could convert the local time to UTC and then get the timestamp from UTC time.

There are two main issues:

• local time may be non-existent or ambiguous e.g. during DST transitions the same time may occur twice
• UTC offset for the local timezone may be different in the past and therefore a naive: local time minus epoch in local time formula may fail

Both can be solved using the tz database (pytz module in Python):

from datetime import datetime
import pytz # $ pip install pytz
from tzlocal import get_localzone # $ pip install tzlocal

tz = get_localzone() # get pytz timezone corresponding to the local timezone

naive_d = datetime.strptime(myDate, "%Y-%m-%d %H:%M:%S,%f")
# a) raise exception for non-existent or ambiguous times
d = tz.localize(naive_d, is_dst=None)
## b) assume standard time, adjust non-existent times
#d = tz.normalize(tz.localize(naive_d, is_dst=False))
## c) assume DST is in effect, adjust non-existent times
#d = tz.normalize(tz.localize(naive_d, is_dst=True))
timestamp = d - datetime(1970, 1, 1, tzinfo=pytz.utc)

The result is timestamp -- a timedelta object, you can convert it to seconds, milliseconds, etc.

Also different systems may behave differently around/during leap seconds. Most application can ignore that they exist.

In general, it might be simpler to store POSIX timestamps in addition to the local time instead of trying to guess it from the local time.