azure-data-factory-2

问题 I have an ADF pipeline that uses copy data activity for copying data from blob storage to table storage. This pipeline runs on a trigger once every day. I have provided a table name in table storage data set as 'Table1'. Instead of providing a hard coded table name value (Table1), is it possible to provide a dynamic value as table name in the table storage such that RUN ID of pipeline run is used as the table name in the table storage and copy data from blob to that table in table storage?

问题 Using azure data factory v2 with GIT / Azure DevOps integration: If you for example create a trigger using Set-AzDataFactoryV2Trigger via powershell according to the documentation, the trigger is created directly in the adf_publish branch. This is an issue, as this will result in a mismatch between the master branch and adf_publish, meaning you'll not be able to publish going forward as this of course raises an error. How do I get the cmdlet to create the trigger in a new or specific branch,

问题 Using azure data factory v2 with GIT / Azure DevOps integration: If you for example create a trigger using Set-AzDataFactoryV2Trigger via powershell according to the documentation, the trigger is created directly in the adf_publish branch. This is an issue, as this will result in a mismatch between the master branch and adf_publish, meaning you'll not be able to publish going forward as this of course raises an error. How do I get the cmdlet to create the trigger in a new or specific branch,

问题 I am setting up an ADF pipeline to copy blob into an Azure SQL DB. I have a Iteration activity in my pipeline, where I have set up a counter to loop and copy only if the blob exists. This works great except for some random PK violations, which I will have to check manually. So I edited my pipeline to log the error, and continue. So I set up the pipeline as such. If the copy activity fails due to Primary Key Violation, (for now) ignore, but log the details using a stored procedure and continue

问题 I am setting up an ADF pipeline to copy blob into an Azure SQL DB. I have a Iteration activity in my pipeline, where I have set up a counter to loop and copy only if the blob exists. This works great except for some random PK violations, which I will have to check manually. So I edited my pipeline to log the error, and continue. So I set up the pipeline as such. If the copy activity fails due to Primary Key Violation, (for now) ignore, but log the details using a stored procedure and continue

问题 There is NO Sql Script activity in Azure Data Factory V2. So how can I create a stored proc, a schema in a database? What are my options? 回答1: There is a preCopyScript property. You could put your script there. It will be executed before each run. You could use store procedure activity as Summit mentioned. You could also create a custom activity. 回答2: I agree that the absence of something like "Execute SQL task" is SSIS is bumming. I normally use a "LookUp" activity as I don't like to create

问题 I have complex XML files with nested elements. I built a process to handle using SSIS and T-SQL. We utilize Azure Data Factory and I'd like to explore converting XML files to JSON or CSV, since those are supported by ADF and XML is not. It appears logic apps is one option. Has anyone had other luck with taking XML and converting within a pipeline? Current Workflow: pick up XML files from folder, drop to on network drives, bulk insert XML into a staging row, parse XML to various SQL tables for

问题 I have complex XML files with nested elements. I built a process to handle using SSIS and T-SQL. We utilize Azure Data Factory and I'd like to explore converting XML files to JSON or CSV, since those are supported by ADF and XML is not. It appears logic apps is one option. Has anyone had other luck with taking XML and converting within a pipeline? Current Workflow: pick up XML files from folder, drop to on network drives, bulk insert XML into a staging row, parse XML to various SQL tables for

问题 I am having a CSV file in my Azure Blob Storage which contain all the column name with its data Data type of respective tables. I want to create a table in SQL Database from this Blob file with the same column name with its corresponding datatype without doing the mapping. I have created a table through data flow but I have to set the data type of each column manually. But I don't want to do this. When I create a table it should accept the same data types in the source as well as synch which

问题 I have an input JSON file where the actual value of the property could be either a numeric value or a string.I extract the value by specifying a json path expression like "fieldValue": "values[*].value" in the azure data factory copy activity, connection tab for the source. Since the actual field value in the JSON could be something like "X" or 2.34 it is not able parse it all into strings even though in the schema I specify the fieldValue as string. So is there a way I could cast it so that