azure-data-factory

问题 Using azure data factory v2 with GIT / Azure DevOps integration: If you for example create a trigger using Set-AzDataFactoryV2Trigger via powershell according to the documentation, the trigger is created directly in the adf_publish branch. This is an issue, as this will result in a mismatch between the master branch and adf_publish, meaning you'll not be able to publish going forward as this of course raises an error. How do I get the cmdlet to create the trigger in a new or specific branch,

问题 I'm trying to build a very basic data flow in Azure Data Factory pulling a JSON file from blob storage, performing a transformation on some columns, and storing in a SQL database. I originally authenticated to the storage account using Managed Identity, but I get the error below when attempting to test the connection to the source: com.microsoft.dataflow.broker.MissingRequiredPropertyException: account is a required property for [myStorageAccountName]. com.microsoft.dataflow.broker

问题 I'm trying to build a very basic data flow in Azure Data Factory pulling a JSON file from blob storage, performing a transformation on some columns, and storing in a SQL database. I originally authenticated to the storage account using Managed Identity, but I get the error below when attempting to test the connection to the source: com.microsoft.dataflow.broker.MissingRequiredPropertyException: account is a required property for [myStorageAccountName]. com.microsoft.dataflow.broker

问题 I was requested to build an ETL pipeline in Azure. This pipeline should read ORC file submitted by the vendor to ADLS parse the PARAMS field, existing in the ORC structure, where JSON structure is stored, and add it as two new fields (KEY, VALUE) to the output write the output to the Azure SQL database The problem is, that there are different types of JSONs structures used by the different types of records. I do not want to write a custom expression per each of the class of JSON struct (there

问题 I was requested to build an ETL pipeline in Azure. This pipeline should read ORC file submitted by the vendor to ADLS parse the PARAMS field, existing in the ORC structure, where JSON structure is stored, and add it as two new fields (KEY, VALUE) to the output write the output to the Azure SQL database The problem is, that there are different types of JSONs structures used by the different types of records. I do not want to write a custom expression per each of the class of JSON struct (there

问题 I'm trying to implement a pipeline on azure data-factory where a I get access to datas from ODATA. Taken these I implement a pipeline where azure DF copies these on a blob storage. For what concerns access and permissions there is no problem. The issue rises up when I want to get a preview of what I'm going to store inside my blob storage. The reported error is: A supported MIME type could not be found that matches the content type of the response. None of the supported type(s) ... Don't ask

问题 I have an input JSON file where the actual value of the property could be either a numeric value or a string.I extract the value by specifying a json path expression like "fieldValue": "values[*].value" in the azure data factory copy activity, connection tab for the source. Since the actual field value in the JSON could be something like "X" or 2.34 it is not able parse it all into strings even though in the schema I specify the fieldValue as string. So is there a way I could cast it so that

问题 I have a very simple ADF pipeline to copy data from local mongoDB (self-hosted integration environment) to Azure SQL database. My pipleline is able to copy the data from mongoDB and insert into SQL db. Currently if I run the pipeline it inserts duplicate data if run multiple times. I have made _id column as unique in SQL database and now running pipeline throws and error because of SQL constraint wont letting it insert the record. How do I check for duplicate _id before inserting into SQL db?

问题 I am using Microsoft.Azure.Management.DataFactories .net core package. I am using following code to get the required token for accessing azure data factory pipeline in C# .net core : public static void RunDataFactoryPipeline() { try { var context = new AuthenticationContext("" + ""); var credentials = new ClientCredential(clientId: "", clientSecret: ""); AuthenticationResult result = context.AcquireTokenAsync("", credentials).Result; if (result == null) { throw new InvalidOperationException(

问题 I am copying records from an Azure Storage Table (source) to Azure Storage Table (sink) everyday. So if I am executing the query on December 24th 2019 (UTC) for instance, then I want to copy records for December 23rd 2019 (UTC). The query works and is doing what I intend it to do. Here is the query: Timestamp ge datetime'2019-12-23T00:00Z' and Timestamp lt datetime'2019-12-24T00:00Z' In the query above, the Timestamp column is automatically stamped in the Azure Storage Table when a new record