azure-data-factory

Azure data factory and stored procedure

由 不想你离开。 提交于 2019-12-08 09:35:11
问题 I've got problem with Azure Data Factory and Stored Procedure. I've set SP as sink for input data: "sink": { "type": "SqlSink", "sqlWriterStoredProcedureName": "spAddProducts", "storedProcedureParameters": { "stringProductData": { "value": "str1" } }, and after execution I've got to process about 200k records, but after some limited number of processed rows (about 10k), I've got error: Copy activity met invalid parameters: ErrorCode=InvalidParameter,'Type=Microsoft.DataTransfer.Common.Shared

Loading data from Excel file into Azure Datawarehouse

由 本小妞迷上赌 提交于 2019-12-08 06:42:22
问题 I have succeeded in loading data from CSV to Azure SQL Server data warehouse, however I now have a excel file as a source and when I try reading this excel file into BLOB CSV it creates a file with junk characters. Any help would be appreciated. 回答1: To handle this within data factory you'll need to use a custom activity (DotNotActivity) that first converts the Excel file to CSV. Then have a downstream activity that deals with the CSV dataset as required. The custom activity will require some

Loading data from Excel file into Azure Datawarehouse

由 感情迁移 提交于 2019-12-08 05:35:25
I have succeeded in loading data from CSV to Azure SQL Server data warehouse, however I now have a excel file as a source and when I try reading this excel file into BLOB CSV it creates a file with junk characters. Any help would be appreciated. To handle this within data factory you'll need to use a custom activity (DotNotActivity) that first converts the Excel file to CSV. Then have a downstream activity that deals with the CSV dataset as required. The custom activity will require some C# classes to be written that handle the conversation. Either using the Office Interoperability libraries

Azure Data Factory - can't convert from “null” to datetime field

由 流过昼夜 提交于 2019-12-07 18:35:19
问题 I have an Azure data factory pipeline which defines data imports from CSV files to SQL server database tables. Some of the tables have nullable datetime fields and the CSV files supply nulls as "null" (i.e. within quotes). However, when I run the pipeline, I'm getting several errors failing to convert 'null' to datetime. I've checked the Azure documentation which states that you can define how null values are presented in the CSV file. Currently, I have the following configuration: "type":

Bringing incremental data in from REST APIs into SQL azure

由 懵懂的女人 提交于 2019-12-07 15:41:29
问题 My needs are following: - Need to fetch data from a 3rd party API into SQL azure. The API's will be queried everyday for incremental data and may require pagination as by default any API response will give only Top N records. The API also needs an auth token to work, which is the first call before we start downloading data from endpoints. Due to last two reasons, I've opted for Function App which will be triggered daily rather than data factory which can query web APIs. Is there a better way

How can we create Azure's Data Factory pipeline with Cosoms DB (with Graph API) as data sink ?

由 老子叫甜甜 提交于 2019-12-07 12:08:25
问题 How can we create Azure's Data Factory pipeline with Cosoms DB (with Graph API) as data sink ? (data source being Cosmos DB only (Document DB as API) 回答1: Cosmos DB Graph API is not supported yet and we will add to our product backlog. 回答2: One option that is available to you is to simply continue using the Document API for the graph enabled CosmosDB sink. If you transform and write your documents into the destination in GraphSON format as regular documents they will be automatically usable

Azure Data flow taking mins to trigger next pipeline

由 痞子三分冷 提交于 2019-12-07 11:39:17
Azure Data factory transferring data in Db in 10 millisecond but the issue I am having is it is waiting for few mins to trigger next pipeline and that ends up with 40 mins all pipelines are taking less than 20 ms to transfer data. But somehow it is waiting a few mins to trigger the next one. I used debug mode as well trigger the ADF using Logic App without debugging mood. Is there any way I can optimize it we want to move from SSIS to Data Flow but having a time issue 40 mins are so much in next step we have millions of records so it took 7 seconds to transfer data to dataBase but it waited

How to read files with .xlsx and .xls extension in Azure data factory?

由 走远了吗. 提交于 2019-12-07 04:44:15
问题 I am trying to read and excel file in Azure Blob Storage with .xlsx extension in my azure data factory dataset. it throws following error Error found when processing 'Csv/Tsv Format Text' source 'Filename.xlsx' with row number 3: found more columns than expected column count: 1. What are the right Column and row delimiters for excel files to be read in azure Data factory 回答1: Excel files have a proprietary format and are not simple delimited files. As indicated here, Azure Data Factory does

How to setup custom Spark parameter in HDInsights cluster with Data Factory

由 馋奶兔 提交于 2019-12-06 11:50:54
I am creating HDInsights cluster on Azure according to this desciption Now I would like to set up spark custom parameter, for example spark.yarn.appMasterEnv.PYSPARK3_PYTHON or spark_daemon_memory in time of cluster provisioning. Is it possible to setup using Data Factory/Automation Account? I can not find any example doing this. Thanks You can use SparkConfig in Data Factory to pass these configurations to Spark. For example: "typeProperties": { ... "sparkConfig": { "spark.submit.pyFiles": "/dist/package_name-1.0.0-py3.5.egg", "spark.yarn.appMasterEnv.PYSPARK_PYTHON": "/usr/bin/anaconda/envs

Referencing JSON payload value in Azure Data Factory for If condition

由 ぐ巨炮叔叔 提交于 2019-12-06 10:35:38
I have a Json file like so as a payload returned from an API call - which is an http dataset type in data factory. { "count": 2, "name": "DatasetABC", "columnNames": [ "Column_1", "Column_2" ], "rows": ["1234", "5678" ] } I would like to be able to use the count records returned in an If condition. Im wondering what I need to use to get the value of "count" which is 2. Any help appreciated. Based on your description, i suppose you could use LookUp Activity in Azure Data Factory. Lookup activity can retrieve a dataset from any of the Azure Data Factory-supported data sources. Use it in the
订阅 azure-data-factory