azure-data-factory

问题 Customer 's requirements: using use azure data factory to import csv file in blob storage to SQL data warehouse. using the strategy "Fault tolerance and log the incompatible rows in Azure Blob storage" in ADF. And using Azure Function to archive the processed file to other place in blob storage: one place for those files are imported successfully and one for fail files (the files have incompatible data - wrong format, wrong length) => so I need get value of skippedRowCount of Activity Window

问题 I have multiple csv files in my azure blob storage which I wish to append into one csv file also stored in azure blob storage using the azure data factory pipeline. The problem is that all the columns of the source files are not present in the sink file and vice versa and also all the source files are not identical. I just want to map the columns I need from source files to the columns in sink file. The copy activity in the data factory is not allowing me to do so. 回答1: According my

问题 I have a for each activity which has a stored procedure (SP) wherein I am inputing values using item() evidently. Now suppose SP's input values are item().a, item().b and item().c Question: For some of the iteration of foreach, item().b does not exist which is expected. So how should i deal with it in the Stored procedure? Because at this point of time it is giving me an error when it executed SP by saying: "The template language expression 'item().b' cannot be evaluated because property 'b'

问题 I am trying to copy data from Table storage to another Table storage of a different storage account, for that, I am using Copy activity in Azure data factory. I want to filter rows that are going to be copied to the sink table storage, for that Azure data factory gives an option to define a query. I want to apply a filter on the Partition key whose datatype is String but holds numeric values. I am looking at this documentation: https://docs.microsoft.com/en-us/azure/devops/pipelines/process

问题 I'm trying to copy data from a view in either on-premise PostgeSQL or MSSQL to a table in Azure PostgreSQL. I can't seem to get this to successfully copy when I map more than 6 columns from source to sink. I suspected that one of the source columns was the issue, so varied what columns were mapped but all columns will succeed if less than 6 are copied in total. I then tried different source and sink tables, which produces the same results. If I copy to an intermediary csv file I can import

问题 I have PowerShell script which splits a complex CSV file to a smaller CSV file for every 1000 records. Here is the code: $i=0;Get-Content C:\Users\dell\Desktop\Powershell\Input\bigsizeFile.csv -ReadCount 1000 | %{$i++; $_ | Out-File C:\Users\dell\Desktop\Powershell\Output\file$i.csv } Now I want to use this script in Azure PowerShell and I want to run this from Azure Data Factory. Can Someone please help with this. 回答1: you could execute your powershell command by using Custom activity in

问题 I am trying to enable diagnostics for an azure datafactory using an ARM Template after is has been created via a c# automation application. I am attempting to use the steps here for the Non-Compute Resource Template: https://docs.microsoft.com/en-us/azure/monitoring-and-diagnostics/monitoring-enable-diagnostic-logs-using-template Step 2 of the above instructions state: In the resources array of the resource for which you want to enable Diagnostic Logs, add a resource of type [resource

问题 I am using Data factory to create our staging area, the problem is whenever source data changes, we add a new row to staging tables. For instance, assume we have the following data: ID Fields created edited 100 ---------- '2017-07-01' '2017-07-05' this will be stored in our staging tables like this: ID Fields created edited 100 ---------- '2017-07-01' null 100 ---------- '2017-07-01' '2017-07-05' Selecting the most recent row is expensive and We don't want that. How do you think we can avoid

问题 I have a c sharp class library that connects to Azure Analysis Services using the AMO library. I'd like to use this as part of my data factory pipeline to refresh cube partitions. This is done through Azure batch as a custom .net activity. var server = new Server(); server.Connect("Provider=MSOLAP;Data Source=asazure://uksouth.asazure.windows.net/abcd;Initial Catalog=xyz"); Running this locally works fine, however this will not run in the cloud. It currently errors out as it is not being run

问题 I am using Azure Data Catalog of my organization. I am not creator/administrator/owner of the Catalog but I have access to register/delete catalogs from the web interface. I want to use rest API for Azure Data Catalog. Is it possible with my level of permission? I have followed all the steps from https://msdn.microsoft.com/en-us/library/mt428033.aspx and written the following piece of code: class Program { static void Main(string[] args) { string url = "https://api.azuredatacatalog.com