azure-data-factory

问题 I am trying to copy data from Azure CosmosDB into Azure SQL Database. In the Azure CosmosDB source, the query is as follows : select * from c where c.DefinitionTime >= '@{formatDateTime(pipeline().parameters.windowStart, 'yyyy-MM-ddTHH:mm:ssZ' )}' AND c.DefinitionTime < '@{formatDateTime(pipeline().parameters.windowEnd, 'yyyy-MM-ddTHH:mm:ssZ' )}' In CosmosDB, "DefinitionTime" is stored as a string. To the run the above query, I have to update "DefinitionTime" with range-indexing which is one

问题 We are moving the data from Azure Table Storage to Azure Data Lake Store with the help of Data Factory. As part of if I have added a linked service which connects to Azure Data Lake Store. { "name": "AzureDataLakeStoreLinkedService", "properties": { "description": "", "hubName": "xxxxxxxxxxxxxxxxxxxxxx", "type": "AzureDataLakeStore", "typeProperties": { "dataLakeStoreUri": "https://xxxxxxxxxxxxx.azuredatalakestore.net/webhdfs/v1", "authorization": "**********", "sessionId": "**********",

问题 I have a Data Lake Store account. I have a directory full of files containing data in JSON format, including some string values that contain times in ISO 8601 format, to wit: { "reading_time": "2008-09-15T15:53:00.91077" } Now when I create a Pipeline with a Data Factory that uses these JSON files as an input dataset, it sees the value of reading_time in a typical US format: "9/15/2008 3:53:00 PM" . Specifically, I get this message when I try to populate a DateTime field in the output dataset

问题 I'm a SSIS Developer. I do lots of SQL stored procedure lookup concepts in SSIS. But when coming to Azure Data Factory I haven't any idea how to perform a lookup using a SQL stored procedure. Could anyone please guide me on this? Thanks in advance ! Jay 回答1: Azure Data Factory (ADF) is more of an ELT tool rather than ETL, therefore direct lookups are not supported. Instead, this type of operation, along with other transforms is pushed down into the compute you are actually using. For example,

问题 So I am testing each of the activities of the pipeline and I want to disable some of the activities in it. Essentially there is an activity of sending emails which I want to disable as I wanted to see the output of prior activities. Offcourse I dont want to remove the email sending activity because it is in the prod environment and not developed by me. Is there any way to disable it? 回答1: You cannot disable one, but what you want to do is possible with the debug option in the editor. Just

问题 I am looking at the link below. https://azure.microsoft.com/en-us/updates/data-factory-supports-wildcard-file-filter-for-copy-activity/ We are supposed to have the ability to use wildcard characters in folder paths and file names. If we click on the 'Activity' and click 'Source', we see this view. I would like to loop through months any days, so it should be something like this view. Of course that doesn't actually work. I'm getting errors that read: ErrorCode: 'PathNotFound'. Message: 'The

问题 I have a table into an Azure Databricks Cluster, i would like to replicate this data into an Azure SQL Database, to let another users analyze this data from Metabase. Is it possible to acess databricks tables through Azure Data factory? 回答1: No, unfortunately not. Databricks tables are typically temporary and last as long as your job/session is running. See here. You would need to persist your databricks table to some storage in order to access it. Change your databricks job to dump the table

问题 I am creating HDInsights cluster on Azure according to this desciption Now I would like to set up spark custom parameter, for example spark.yarn.appMasterEnv.PYSPARK3_PYTHON or spark_daemon_memory in time of cluster provisioning. Is it possible to setup using Data Factory/Automation Account? I can not find any example doing this. Thanks 回答1: You can use SparkConfig in Data Factory to pass these configurations to Spark. For example: "typeProperties": { ... "sparkConfig": { "spark.submit

问题 I am making a custom activity in ADF, which involves reading multiple files from Azure Storage Blobs, doing some work on them, and then finally writing a resulting file to the Azure Data Lake Store. Last step is where I stop, because as far as I can see, the .NET SDK only allows for uploading from a local file. Is there any way to to (programmatically) upload a file to ADL Store, where it is not from a local file? Could be a blob or a stream. If not, any workarounds? 回答1: Yes, it's possible

问题 I am not sure whether Azure Data Factory project is supported on Visual Studio 2017 at the moment. I have just installed VS 2017 but cannot open our solution as there is one azure data factory project. Does Azure Data Factory support on Visual Studio 2017? 回答1: I tweeted the Visual Studio team with exactly this issue a couple of days ago. I even included a screen shot very similar to yours. This is the response I got: @mrpaulandrew We have no info about it yet. Our team is already working on