Parsing of Ordered Timestamps in Local Time (to UTC) While Observing Daylight Saving Time

Question

I have CSV data files with timestamped records that are in local time. Unfortunately the data files cover the period where daylight saving time changes (Nov 3rd 2013) so th

Answer 1

If successive timestamps can't go backwards if expressed as time in UTC then this Python script can convert the local time into UTC:

#!/usr/bin/env python3
import sys
from datetime import datetime, timedelta
import pytz  # $ pip install pytz

tz = pytz.timezone('America/New_York' if len(sys.argv) < 2 else sys.argv[1])
previous = None #XXX set it from UTC time: `first_entry_utc.astimezone(tz)`
for line in sys.stdin: # read from stdin
    naive = datetime.strptime(line.strip(), "%Y/%m/%d %H:%M:%S") # no timezone
    try:
        local = tz.localize(naive, is_dst=None) # attach timezone info
    except pytz.AmbiguousTimeError:
        # assume ambiguous time always corresponds to True -> False transition
        local = tz.localize(naive, is_dst=True)
        if previous >= local: # timestamps must be increasing
            local = tz.localize(naive, is_dst=False)
        assert previous < local
    #NOTE: allow NonExistentTimeError to propagate (there shouldn't be
    # invalid local times in the input)
    previous = local
    utc = local.astimezone(pytz.utc)
    timestamp = utc.timestamp()
    time_format = "%Y-%m-%d %H:%M:%S %Z%z"
    print("{local:{time_format}}; {utc:{time_format}}; {timestamp:.0f}"
          .format_map(vars()))

Input

2013/11/03 00:45:00
2013/11/03 01:00:00
2013/11/03 01:15:00
2013/11/03 01:30:00
2013/11/03 01:45:00
2013/11/03 01:00:00
2013/11/03 01:15:00
2013/11/03 01:30:00
2013/11/03 01:45:00
2013/11/03 02:00:00

Output

2013-11-03 00:45:00 EDT-0400; 2013-11-03 04:45:00 UTC+0000; 1383453900
2013-11-03 01:00:00 EDT-0400; 2013-11-03 05:00:00 UTC+0000; 1383454800
2013-11-03 01:15:00 EDT-0400; 2013-11-03 05:15:00 UTC+0000; 1383455700
2013-11-03 01:30:00 EDT-0400; 2013-11-03 05:30:00 UTC+0000; 1383456600
2013-11-03 01:45:00 EDT-0400; 2013-11-03 05:45:00 UTC+0000; 1383457500
2013-11-03 01:00:00 EST-0500; 2013-11-03 06:00:00 UTC+0000; 1383458400
2013-11-03 01:15:00 EST-0500; 2013-11-03 06:15:00 UTC+0000; 1383459300
2013-11-03 01:30:00 EST-0500; 2013-11-03 06:30:00 UTC+0000; 1383460200
2013-11-03 01:45:00 EST-0500; 2013-11-03 06:45:00 UTC+0000; 1383461100
2013-11-03 02:00:00 EST-0500; 2013-11-03 07:00:00 UTC+0000; 1383462000