Convert datetime to another format without changing dtype

Question

I'm just learning Pandas myself and I have met few problems.

1. In a DataFrame, which it was reads from a csv file, I have one column includes date data that in different format(like '%m/%d/%Y' and '%Y-%m-%d', may be blank.) and I want to unify the format of this column. But I don't know if there are any other formats. So when I using pd.to_datetime(),it raised some errors like format not matching and not timelike data. How can I unify the format of this column?

2. I have converted part of that column into datetime dtype, and it's in YYYY-mm-dd format. Can I keep the datetime dtype, and change the format into '%m/%d/%Y'? I have used pd.dt.strftime(), it will change the format, but also change the dtype into str, not keeping the datetime dtype.

Answer 1

So when I using pd.to_datetime(),it raised some errors like format not matching and not timelike data. How can I unify the format of this column?

Use the errors='coerce' option in order to return NaT (Not a Time) for non-converted values. Also note that the format argument is not required. Omitting it will enable Pandas to try multiple formats, failing which it will revert to NaT¹. For example:

df['datetime'] = pd.to_datetime(df['datetime'], errors='coerce')

Beware, mixed types may be interpreted incorrectly. For example, how will Python know whether 05/06/2018 is 5th June or 6th May? An order of conventions will be applied and if you need greater control you will need to apply a customised ordering yourself.

Can I keep the datetime dtype, and change the format into '%m/%d/%Y'?

No, you cannot. datetime series are stored internally as integers. Any human-readable date representation is just that, a representation, not the underlying integer. To access your custom formatting, you can use methods available in Pandas. You can even store such a text representation in a pd.Series variable:

formatted_dates = df['datetime'].dt.strftime('%m/%d/%Y')

The dtype of formatted_dates will be object, which indicates that the elements of your series point to arbitrary Python times. In this case, those arbitrary types happen to be all strings.

Lastly, I strongly recommend you do not convert a datetime series to strings until the very last step in your workflow. This is because as soon as you do so, you will no longer be able to use efficient, vectorised operations on such a series.

---

¹ This will sacrifice performance and contrasts with datetime.strptime, which requires format to be specified. Internally, Pandas uses the dateutil library, as indicated in the docs.