azure-data-factory

My needs are following: - Need to fetch data from a 3rd party API into SQL azure. The API's will be queried everyday for incremental data and may require pagination as by default any API response will give only Top N records. The API also needs an auth token to work, which is the first call before we start downloading data from endpoints. Due to last two reasons, I've opted for Function App which will be triggered daily rather than data factory which can query web APIs. Is there a better way to do this? Also I am thinking of pushing all JSON into Blob store and then parsing data from the JSON

Are there any automatic testing mechanism available for azure data factory pipelines? Does the azure data factory visual studio project come with any test suite of its own? Any help highly appreciated Thanks Not that I'm aware of, but happy to be told otherwise. I suggest you post this on Microsoft's user voice page as a feedback idea. Then people searching will come here, go to that link and vote to get something developed. https://feedback.azure.com/forums/270578-data-factory/filters/my_feedback?query=Unit%20Testing%20for%20ADF%20Projects Hope this helps. EDIT after comment: You could use a

I am trying to read and excel file in Azure Blob Storage with .xlsx extension in my azure data factory dataset. it throws following error Error found when processing 'Csv/Tsv Format Text' source 'Filename.xlsx' with row number 3: found more columns than expected column count: 1. What are the right Column and row delimiters for excel files to be read in azure Data factory Excel files have a proprietary format and are not simple delimited files. As indicated here , Azure Data Factory does not have a direct option to import Excel files, eg you cannot create a Linked Service to an Excel file and

I'm trying to parse below Json file using USQL but keep getting error. Json file@ {"dimBetType_SKey":1,"BetType_BKey":1,"BetTypeName":"Test1"} {"dimBetType_SKey":2,"BetType_BKey":2,"BetTypeName":"Test2"} {"dimBetType_SKey":3,"BetType_BKey":3,"BetTypeName":"Test3"} Below is the USQL script, I'm trying to extract the data from above file. REFERENCE ASSEMBLY [Newtonsoft.Json]; REFERENCE ASSEMBLY [Microsoft.Analytics.Samples.Formats]; DECLARE @Full_Path string = "adl://xxxx.azuredatalakestore.net/2017/03/28/00_0_66ffdd26541742fab57139e95080e704.json"; DECLARE @Output_Path = "adl://xxxx

Background : I have scheduled pipelines running for copying data from source to destination. This is scheduled to run daily at a specific time. Problem : The input dataset to the pipeline is external and not available at specific time intervals. This means the copy activity will have to wait until the Scheduled Start time mentioned in the Pipeline to kickoff. Considering the volume of data, I don't want to waste my time here. Requirement : At any given time I have access to the time when my input data set is available. With this in hand, I want to know how to trigger a ADF Pipeline from C#

I have a c sharp class library that connects to Azure Analysis Services using the AMO library. I'd like to use this as part of my data factory pipeline to refresh cube partitions. This is done through Azure batch as a custom .net activity. var server = new Server(); server.Connect("Provider=MSOLAP;Data Source=asazure://uksouth.asazure.windows.net/abcd;Initial Catalog=xyz"); Running this locally works fine, however this will not run in the cloud. It currently errors out as it is not being run under my user account. I know that I can add a username and password to the connection string, but I

When using a Data Factory Copy Activity to take delimited text files from Blob Storage to Table Storage, I am able to specify the input and output schemas and the relevant one-to-one Column Mappings. It is possible to concatenate or format one or more columns (with $$Text.Format perhaps?) in the source schema and map these to a single column in the destination? If possible, this would be especially useful if I can also set the Partition Key using a similar approach (a composite key taken from 2 or more source column values or better still, the SliceStart plus a concatenated source column value

I am making a custom activity in ADF, which involves reading multiple files from Azure Storage Blobs, doing some work on them, and then finally writing a resulting file to the Azure Data Lake Store. Last step is where I stop, because as far as I can see, the .NET SDK only allows for uploading from a local file. Is there any way to to (programmatically) upload a file to ADL Store, where it is not from a local file? Could be a blob or a stream. If not, any workarounds? Yes, it's possible to upload from Stream, the trick is to create file first and then append your stream to it: string