azure-data-factory-2

问题 This is what I followed to setup IR. In the final step of Registering Azure Data factory self hosted intergration runtime, we need to provide the Authentication Key. then the installation is making a call to internet. Isn't this strange as the VM could be in a private network? If the VM is not connected to internet and it gets this error then what to do? "Failed to connect to: we.frontend.clouddatahub.net" This is the error I get Failed to execute the command ' -Key xxx'. Error message:

问题 I have a very simple ADF pipeline to copy data from local mongoDB (self-hosted integration environment) to Azure SQL database. My pipleline is able to copy the data from mongoDB and insert into SQL db. Currently if I run the pipeline it inserts duplicate data if run multiple times. I have made _id column as unique in SQL database and now running pipeline throws and error because of SQL constraint wont letting it insert the record. How do I check for duplicate _id before inserting into SQL db?

问题 I'm actually some automation for my ADF. As a part of that, I'm trying to delete all the ADF V2 pipelines. The problem is my pipelines having many references with different pipelines itself. $ADFPipeline = Get-AzDataFactoryV2Pipeline -DataFactoryName $(datafactory-name) -ResourceGroupName $(rg) $ADFPipeline | ForEach-Object { Remove-AzDataFactoryV2Pipeline -ResourceGroupName $(rg) -DataFactoryName $(datafactory-name) -Name $_.name -Force } And most of the time I get the error like The

问题 I'm actually some automation for my ADF. As a part of that, I'm trying to delete all the ADF V2 pipelines. The problem is my pipelines having many references with different pipelines itself. $ADFPipeline = Get-AzDataFactoryV2Pipeline -DataFactoryName $(datafactory-name) -ResourceGroupName $(rg) $ADFPipeline | ForEach-Object { Remove-AzDataFactoryV2Pipeline -ResourceGroupName $(rg) -DataFactoryName $(datafactory-name) -Name $_.name -Force } And most of the time I get the error like The

问题 I need to create a global temp table in my SQL Server while executing an Azure Data Factory pipeline. This table will be used in several activities. I already tried several approaches including one using the Stored Procedure activity targeting the sys.sp_executesql SP and the CREATE TABLE statement as the parameter. With this approach the table is actually created, but it's automaticaly dropped a second later, I don't understand why. This is the script used to create the temp table: CREATE

问题 Very simple issue. I am trying to set up a pipeline that has a variable of type bool. As soon as I add it, the pipeline fails with: { "code":"BadRequest", "message":"Invalid value for property 'type'", "target":"pipeline/pipeline2/runid/66b9c7be-9894-494a-abd9-34fd92bbd972", "details":null, "error":null } simple pipeline with a string variable and a wait activity succeeds. { "name": "pipeline2", "properties": { "activities": [ { "name": "Wait1", "type": "Wait", "typeProperties": {

问题 I have created a pipeline which has two parameters at pipeline level. I want to send the values to these parameters using powershell and trigger the pipeline. Any idea how to do it using Powershell. 回答1: I'll leave a script that you can then modify to your needs: Login-AzureRmAccount Select-AzureRmSubscription -Subscription "yourSubId" $dfname = "youDataFActoryName" $rgName = "yourResourceGroupName" $pipe = "pipeName" $parameters = @{ "param1" = "asdasd" "param2" = "123456" } Invoke

问题 I have created a pipeline which has two parameters at pipeline level. I want to send the values to these parameters using powershell and trigger the pipeline. Any idea how to do it using Powershell. 回答1: I'll leave a script that you can then modify to your needs: Login-AzureRmAccount Select-AzureRmSubscription -Subscription "yourSubId" $dfname = "youDataFActoryName" $rgName = "yourResourceGroupName" $pipe = "pipeName" $parameters = @{ "param1" = "asdasd" "param2" = "123456" } Invoke

问题 This is a generic question regarding development processes and using Azure Data Factory v2. I am currently using the UI portal to set up and configure pipelines, datasets, activities, triggers, etc. but I am finding the lag time very long. Is this the same for other users ? What is the typical workflow for someone not using the portal ? (There seem to be no nuget packages for v2 and only powershell as the alternative to the UI portal.) 回答1: The publish performance is low when you have large

问题 I've got a Get metadata activity that goes to an sftp server and lists the files: Is it possible to output this list to a file without using a function?? 回答1: Is there a particular reason you have to log the output to a file? If you just call a Get Metadata task in Azure Data Factory it will be logged as part of the pipeline run and default logging anyway. You can then access those logs if required. Alternately, a common pattern I use with Get Metadata task is a For Each loop and then host