azure-data-factory

I created a pipeline and two linked services to move data from an on-prem instance of SQL Server to an Azure Sql instance. The issue I'm running into is that we have a table "Table-1" in our on-prem with an Identity (1,1) column that is missing a sequential ID (e.g. the values are 1, 2, 3, 4, 6). When the pipeline runs, it tries to insert the rows with the ID's 1, 2, 3, 4, 5 which is a big problem because ID 6 is a foreign key on another table "Table-2" and now it doesn't exist, so the movement of data to Table-2 fails with SQL Error 547 (Insert statement conflicted with the foreign key

In Azure Data Factory (V1) I was able to create a slide and store the output to a specific folder (i.e. {Year}/{Month}/{Day}. See code below. How do you create the same type of slice in Azure Data Factory V2? I did find that you have to create a paramater. Yes, I was unable to figure out how to pass the parameter. "folderPath": "@{dataset().path}", "parameters": { "path": { "type": "String" Here is original ADF V1 code. { "name": "EMS_EMSActivations_L1_Snapshot", "properties": { "published": false, "type": "AzureDataLakeStore", "linkedServiceName": "SalesIntelligence_ADLS_LS", "typeProperties"

I have a number of files that I transferred into Azure Blob Storage via the Azure Data Factory. Unfortunately, this tool doesn't appear to set the Content-MD5 value for any of the values, so when I pull that value from the Blob Storage API, it's empty. I'm aiming to transfer these files out of Azure Blob Storage and into Google Storage. The documentation I'm seeing for Google's Storagetransfer service at https://cloud.google.com/storage/transfer/reference/rest/v1/TransferSpec#HttpData indicates that I can easily initiate such a transfer if I supply a list of the files with their URL, length in

I'am trying to integrate a WebJob inside an ADF pipeline. The webjob is a very simple console application: namespace WebJob4 { class ReturnTest { static double CalculateArea(int r) { double area = r * r * Math.PI; return area; } static void Main() { int radius = 5; double result = CalculateArea(radius); Console.WriteLine("The area is {0:0.00}", result); } } } How do we call this webjob through an ADF pipeline and store the response code (HTTP 200 in case of Success) in azure blob storage? Dec 2018 Update : If you are thinking of doing this using azure function, azure data factory NOW provides

Going through this tutorial "Create a pipeline with Copy Activity using Visual Studio" and recieving this error when i hit publish. Creating datafactory-Name:VSTutorialFactory,Tags:,Subscription:Pay-As-You-Go,ResourceGroup:MyAppGroup,Location:North Europe, 24/03/2016 11:30:34- Error creating data factory: Microsoft.WindowsAzure.CloudException: MissingSubscriptionRegistration: The subscription is not registered to use namespace 'Microsoft.DataFactory'. Error not mentioned anywhere on net and very little help/knowledge on azure generally on web. In Azure, for each functionality there's a

I created a (once run) DF (V2) pipeline to load files (.lta.gz) from a SFTP server into an azure blob to get historical data. Worked beautifully. Every day there will be several new files on the SFTP server (which cannot be manipulated or deleted). So I want to create an incremental load pipeline which checks daily for new files - if so ---> copy new files. Does anyone have any tips for me how to achieve this? Thanks for using Data Factory! To incrementally load newly generated files on SFTP server, you can leverage the GetMetadata activity to retrieve the LastModifiedDate property: https:/