azure-data-factory

问题 I keep getting following error from Data Factory whenever I run an U-SQL Job Job submission failed, the user 'adla account name' does not have permissions to a subfolder in the /system/ path needed by Data Lake Analytics. Please run “Add User Wizard” from the Data Lake Analytics Azure Portal or use Azure PowerShell to grant access for the user to the /system/ and its children on the Data Lake Store. And I am not using any Firewall as suggested in this post: Run U-SQL Script from C# code with

问题 I have 2 blob files to copy to Azure SQL tables. My pipeline with two activities: { "name": "NutrientDataBlobToAzureSqlPipeline", "properties": { "description": "Copy nutrient data from Azure BLOB to Azure SQL", "activities": [ { "type": "Copy", "typeProperties": { "source": { "type": "BlobSource" }, "sink": { "type": "SqlSink", "writeBatchSize": 10000, "writeBatchTimeout": "60.00:00:00" } }, "inputs": [ { "name": "FoodGroupDescriptionsAzureBlob" } ], "outputs": [ { "name":

问题 I have a datafactory that reads from a table and stores the output as a CSV to Blob Storage. I have noticed that instead of leaving a NULL field blank it inserts the NULL character \N.. Now the external system that is ingesting this can't handle \N. Is there anyway in my dataset where I can say leave nulls blank. Below is my dataset properties: "typeProperties": { "fileName": "MasterFile-{fileDateNameVariable}.csv", "folderPath": "master-file-landing", "format": { "type": "TextFormat",

问题 This is a double post with MSDN, but didn't get any help there so i'm hoping some expert sees it here. I started from the example found at https://docs.microsoft.com/en-us/azure/data-factory/tutorial-incremental-copy-powershell "name": "SinkDataset", "properties": { "type": "AzureBlob", "typeProperties": { "folderPath": "adftutorial/incrementalcopy", "fileName": "@CONCAT('Incremental-', pipeline().RunId, '.txt')", "format": { "type": "TextFormat" } }, My code became "typeProperties": {

问题 I have a U-SQL managed table that contains schematized structured data. CREATE TABLE [AdlaDb].[dbo].[User] ( UserGuid Guid, Postcode string, Age int? DateOfBirth DateTime?, ) And a Azure SQL Database table. CREATE TABLE [SqlDb].[dbo].[User] ( UserGuid uniqueidentifier NOT NULL, Postcode varchar(15) NULL, Age int NULL, DateOfBirth Date NULL, ) I would like to transfer data from U-SQL managed table to Azure SQLDB table without losing the data types. I'm using azure data factory, seems like I

问题 I am trying to create my Azure DevOps release pipeline for Azure Data Factory. I have followed the rather cryptic guide from Microsoft (https://docs.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment ) regarding adding additional parameters to the ARM template that gets generated when you do a publish (https://docs.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment#use-custom-parameters-with-the-resource-manager-template ) Created a arm-template

问题 I have a csv file as input which I have stored in Azure Blob Storage. I want to read data from csv file, perform some transformations on it and then store data in Azure SQL Database. I am trying to use a C# custom activity in Azure Data Factory having blob as input and sql table as output dataset. I am following this tutorial (https://azure.microsoft.com/en-us/documentation/articles/data-factory-use-custom-activities/#see-also) but it has both input and output as blobs. Can I get some sample

问题 Background : I have scheduled pipelines running for copying data from source to destination. This is scheduled to run daily at a specific time. Problem : The input dataset to the pipeline is external and not available at specific time intervals. This means the copy activity will have to wait until the Scheduled Start time mentioned in the Pipeline to kickoff. Considering the volume of data, I don't want to waste my time here. Requirement : At any given time I have access to the time when my

问题 I have a data factory with multiple pipelines and each pipeline has around 20 copy activities to copy azure tables between 2 storage accounts. Each pipeline handles a snapshot of each azure table hence i want to run pipelines sequentially to avoid the risk of overwriting latest data with old data. I know that giving first pipeline output as input to the 2nd pipeline we can achieve this. But as i have many activities in a pipeline, i am not sure which activity will complete last. Is there

问题 I am exploring ways to incrementally copy S3 blobs to azure blobs. I tried azure data factory and I was not able to find this option. S3 has millions of objects and without an incremental option it takes hours to complete. I am open to explore other tools/options. 回答1: if you need to do daily/hourly incrementally copy, the scheduled/tumbling window trigger should be the option. Here is an example to reference. ADFv2 also supports compression in copy, you could specify the type and level of