azure-data-factory

问题 I am using the copy activity to copy/transform a JSON source dataset into JSON sink dataset. Need to append a few audit fields to the output - such as transform date using @utcnow expression function. How can this be accomplished? 回答1: It looks like the databricks activity handles this functionality pretty well. df_new = df.select("table.field1","table.field2","table.field3").withColumn("TransferDate", current_timestamp()) 来源： https://stackoverflow.com/questions/51883988/data-factory-append

问题 I am creating an extensive data factory work flow that will create and fill a data warehouse for multiple customers automatic, however i'm running into an error. I am going to post the questions first, since the remaining info is a bit long. Keep in mind i'm new to data factory and JSON coding. Questions & comments How do i correctly pass the parameter through to an Execute Pipeline activity? How do i add said parameter to an Azure Function activity? The issue may lie with correctly passing

问题 I have a USQL script stored on my ADL store and I am trying to execute it. the script file is quite big - about 250Mb. So far i have a Data Factory, I have created a Linked Service and am trying to create a Data lake Analytics U-SQL Activity. The code for my U-SQL Activity looks like this: { "name": "RunUSQLScript1", "properties": { "description": "Runs the USQL Script", "activities": [ { "name": "DataLakeAnalyticsUSqlActivityTemplate", "type": "DataLakeAnalyticsU-SQL", "linkedServiceName":

问题 I use Azure Fucntion with Azure SDK and Azure Data Factory, is there any way to get value of skippedRowCount of Activity Window when applying "log the incompatible rows" in Copy Activity (Source: Blob Storage, Sink: SQL Data Warehouse)? 回答1: From the documentation here : Monitor skipped rows, looks like a log file is automatically generated when log incompatible rows is enabled at https://[your-blob-account].blob.core.windows.net/[path-if-configured]/[copy-activity-run-id]/[auto-generated

问题 How do I import a TAR file using Azure Data Factory and unzip (decompress) them into Azure Data lake store. I could find it possible for .gZ and .zip files but not .tar! Just FYI, my .tar file sits in the on-premise file system. Request your kind help. 回答1: You can create a .NET custom activity to install the tar.exe command (that you will have uploaded to Blob store) and run it: https://docs.microsoft.com/en-us/azure/data-factory/data-factory-use-custom-activities You can adapt the following

问题 I have one software installed in Azure VM (window OS) which does analysis on the data. I have 500+ GB data in Azure BLOB storage which needs to be processed by this software. I would like to know if it is possible to attach BLOB storage as a virtual hard disk to this VM OR is it possible to copy data from BLOB storage to Azure File service using Azure Data factory? OR should I keep data in Azure File Service itself so that this software can access data directly without copy operation? 回答1: it

问题 I created a pipeline in data factory and I want to retrieve data from a source for the current month and for the previous month. When I run the pipeline I give the needed parameter named ExtractDate. The format is MM/DD/YYYY . For the current month I used the following expression in 'Set Variable' activity: @replace(item().Query,'EXTRACTDATE',formatDateTime(variables('ExtractDate'), 'yyyyMM')) And for the previous month I tried: @adddays(variables('ExtractDate'),-28) The problem appears when

问题 I am trying to use the GetMetadata activity and a CopyData activity together. My setup looks like this. I am trying to get files names (using GetMetadata) and load these into a field in a SQL Server table (in conjunction with the CopyData). The CopyData works perfectly fine, but I don't see any way to have the GetMetadata get file names and pass those into a field in a table. In my example, I have 4 fields in the source data which match 4 fields in the destination table. The 5th field,

问题 I am now trying to copy data from cosmosdb to data lake store by data factory. However, the performance is poor, about 100KB/s, and the data volume is 100+ GB, and keeps increasing. It will take 10+ days to finish, which is not acceptable. Microsoft document https://docs.microsoft.com/en-us/azure/data-factory/data-factory-copy-activity-performance mentioned that the max speed from cosmos to data lake store is 1MB/s. Even this, the performance is still bad for us. The cosmos migration tool

问题 I am loading SQL server table using ADF and after insertion is over, I have to do little manipulation using below approach Trigger (After insert) - Failed, SQL server not able to detect inserted record that I push using ADF.. **Seems to be a bug** . Stored procedure using user defined table type - Getting error Error Number '156'. Error message from database execution : Incorrect syntax near the keyword 'select'. Must declare the table variable "@a". I have created below pipeline { "name":