Azure Data Factory Source Dataset value from Parameter

问题

I have a Dataset in Azure Datafactory backed by a CSV file. I added an additional column in Dataset and want to pass it's value from Dataset parameter but value never gets copied to the column

  "type": "AzureBlob",
    "structure": 
    [
        {
            "name": "MyField",
            "type": "String"
        }
    ]

I have a defined parameter as well

   "parameters": {
        "MyParameter": {
            "type": "String",
            "defaultValue": "ABC"
        }
    }

How can copy the parameter value to Column? I tried following but doesn't work

   "type": "AzureBlob",
    "structure": 
    [
        {
            "name": "MyField",
            "type": "String",
            "value": "@dataset().MyParameter"
        }
    ]

But this does not work. I am getting NULL in destination although parameter value is set

回答1:

Based on document: Expressions and functions in Azure Data Factory , @dataset().XXX is not supported in Azure Data Factory so far. So, you can't use parameters value as custom column into sink or source with native copy activity directly.

However, you could adopt below workarounds:

1.You could create a custom activity and write code to do whatever you need.

2.You could stage the csv file in a azure data lake, then execute a U-SQL script to read the data from the file and append the new column with the pipeline rundId. Then output it to a new area in the data lake so that the data could be picked up by the rest of your pipeline. To do this, you just need to simply pass a Parameter to U-SQL from ADF. Please refer to the U-SQL Activity.

In this thread: use adf pipeline parameters as source to sink columns while mapping, the customer used the second way.

来源：https://stackoverflow.com/questions/53175753/azure-data-factory-source-dataset-value-from-parameter

标签

azure-data-factory-2