Azure data factory data flow silently NULLing date column

问题

I'm trying to use Azure Data Factory to upsert a CSV into an Azure SQL table. All seemed well until I checked the results. One of the columns is a nullable date. The CSV contains a value like so 1/2/2020 12:00:00 AM. The data flow silently inserts a NULL instead of throwing an error because it didn't like the input. So how can I get my data flow to convert the string to a datetime properly, and then to error out on issues like this in the future? I really don't want silent failures and bad data.

回答1:

The null value is due to incompatible date formats in ADF. You need to do date format conversion.

Is your source date format like this MM/dd/yyyy HH:mm:ss?
If so, you can use Derived column and add the expression toString(toTimestamp(<Your_Column_Name>,'MM/dd/yyyy HH:mm:ss'),'yyyy-MM-dd HH:mm:SS') to format this column to String. It solved the NULL value. Of course you can choose what the date format you want.

I made a test as follows:

1. My data source is from a csv file and the EmpDate is a date type like yours and last row contains a null value.

2. Then I add the expression toString(toTimestamp(EmpDate,'MM/dd/yyyy HH:mm:ss'),'yyyy-MM-dd HH:mm:SS') in the Derived column activity. Here you can choose the date format what you want.

3.According to Mark Kromer's suggestion, I Add Conditional Split directly after the Derived Column and check for isNull(EmpDate). Here I use not(isNull(EmpDate)) expression.

4. In the end, if the EmpDate contains null value, it will go to sink2 else go to sink1. The row contains null value:

来源：https://stackoverflow.com/questions/63625287/azure-data-factory-data-flow-silently-nulling-date-column