azure-data-factory

问题 I have a Json file like so as a payload returned from an API call - which is an http dataset type in data factory. { "count": 2, "name": "DatasetABC", "columnNames": [ "Column_1", "Column_2" ], "rows": ["1234", "5678" ] } I would like to be able to use the count records returned in an If condition. Im wondering what I need to use to get the value of "count" which is 2. Any help appreciated. 回答1: Based on your description, i suppose you could use LookUp Activity in Azure Data Factory. Lookup

问题 Are there any automatic testing mechanism available for azure data factory pipelines? Does the azure data factory visual studio project come with any test suite of its own? Any help highly appreciated Thanks 回答1: Not that I'm aware of, but happy to be told otherwise. I suggest you post this on Microsoft's user voice page as a feedback idea. Then people searching will come here, go to that link and vote to get something developed. https://feedback.azure.com/forums/270578-data-factory/filters

问题 I am considering Google DataFlow as an option for running a pipeline that involves steps like: Downloading images from the web; Processing images. I like that DataFlow manages the lifetime of VMs required to complete the job, so I don't need to start or stop them myself, but all examples I came across use it for data mining kind of tasks. I wonder if it is a viable option for other batch tasks like image processing and crawling. 回答1: This use case is a possible application for Dataflow/Beam.

问题 I'm trying to parse below Json file using USQL but keep getting error. Json file@ {"dimBetType_SKey":1,"BetType_BKey":1,"BetTypeName":"Test1"} {"dimBetType_SKey":2,"BetType_BKey":2,"BetTypeName":"Test2"} {"dimBetType_SKey":3,"BetType_BKey":3,"BetTypeName":"Test3"} Below is the USQL script, I'm trying to extract the data from above file. REFERENCE ASSEMBLY [Newtonsoft.Json]; REFERENCE ASSEMBLY [Microsoft.Analytics.Samples.Formats]; DECLARE @Full_Path string = "adl://xxxx.azuredatalakestore.net

问题 Dynamically changing the connection string for Tablestorage or blob storage in Azure data factory. Currently, I could see such option for database related dataset? How to achieve the same in Table or Blob storage 回答1: In the New Linked service Azure table storage and Click on Advanced and check Specify Dynamic contents in JSON format adf Copy the below JSON to make it Table Storage Parameterize : { "name": "Table", "type": "Microsoft.DataFactory/factories/linkedservices", "properties": {

问题 I've made a pipeline to copy data from one blob storage to another. I want to have incremental copy if it's possible, but haven't found a way to specify it. The reason is I want to run this on a schedule and only copy any new data since last run. 回答1: If your blob name is well named with timestamp, you could follow this doc to copy partitioned data. You could use copy data tool to setup the pipeline. You could select tumbling window and then in file path filed input {year}/{month}/{day}

问题 I have created an Data Factory in Azure with a about 10-15 pipelines in it and they all work fine and I can debug and run them without issue on the Test Data Factory that I have setup. I have also setup Azure DevOps integration with the Data Factory with a CI/CD pipeline. Initially this pipeline worked fine as well and I was able to use the generated ARM template to recreate the Pipelines/Triggers/etc on a production Data Factory instance. Now I have added a few more things to the Data

问题 Can't able to create linked services using Azure data factory (ADF), I have read/write permission for linked services at ADF level. using Microsoft.Azure.Management.ResourceManager; using Microsoft.Azure.Management.DataFactory; using Microsoft.Azure.Management.DataFactory.Models; using Microsoft.IdentityModel.Clients.ActiveDirectory; LinkedServiceResource storageLinkedService = new LinkedServiceResource( new AzureStorageLinkedService { ConnectionString = new SecureString(

问题 The documentation demonstrates how to create a parameter for a connected service but not how to actual pass in that parameter from a dataset or activity. Basically the connection string is coming from a lookup foreach loop and I want to connect to a storage table. The connection looks like this. The test works when passing in a correct parameter: { "name": "StatsStorage", "properties": { "type": "AzureTableStorage", "parameters": { "connectionString": { "type": "String" } }, "annotations": []

问题 I need to be able to copy files and folder from one DataLake to another DataLake on a different subscription, I'm in possession of both Auth Token and secret key. I've tried different solution including: https://medium.com/azure-data-lake/connecting-your-own-hadoop-or-spark-to-azure-data-lake-store-93d426d6a5f4 wich is involving hadoop but didn't worked on two different subscriptions, due to the site-core.xml wich only accept one subscripton. ADLcopy didn't worked as well, neither DataFactory