azure-data-factory

问题 I have a REST API I need to call from Azure Data Factory and insert the data into a SQL table. The format of the JSON returned from the API is in the following format: { "serviceResponse": { "supportOffice": "EUKO", "totalPages": 5, "pageNo": 1, "recordsPerPage": 1000, "projects": [ { "projectID":1 ...} , { "projectID":2 ...} ,... ] } } the URL is in the format http://server.com/api/Projects?pageNo=1 I have managed to set up a RestService to call the API and return the JSON and a SQL Sink

问题 I have a REST API I need to call from Azure Data Factory and insert the data into a SQL table. The format of the JSON returned from the API is in the following format: { "serviceResponse": { "supportOffice": "EUKO", "totalPages": 5, "pageNo": 1, "recordsPerPage": 1000, "projects": [ { "projectID":1 ...} , { "projectID":2 ...} ,... ] } } the URL is in the format http://server.com/api/Projects?pageNo=1 I have managed to set up a RestService to call the API and return the JSON and a SQL Sink

问题 I am thinking of moving our SSIS ETLs to Azure Data Factory. My arguments in favour of such leap are: Our sources and targets are already in the cloud. ADF is cloud native so it seems at good fit. ADF is a service are therefore we could consume and pay for it on demand. SSIS implies licensing costs, and doesn't lend lend it itself naturally for on-demand consumption (we thought of using DevOps to spin ETL servers on an ad-hoc basis) Generating ETL code programmatically with SSIS requires very

问题 I am thinking of moving our SSIS ETLs to Azure Data Factory. My arguments in favour of such leap are: Our sources and targets are already in the cloud. ADF is cloud native so it seems at good fit. ADF is a service are therefore we could consume and pay for it on demand. SSIS implies licensing costs, and doesn't lend lend it itself naturally for on-demand consumption (we thought of using DevOps to spin ETL servers on an ad-hoc basis) Generating ETL code programmatically with SSIS requires very

问题 I want to use Python to add an activity to a pipeline in Azure Data Factory. With the following code I am replacing the actual activity but not adding a new one: p_name = 'test' act_name = 'Wait4' Wait_activity = WaitActivity(name=act_name,wait_time_in_seconds=5) p_obj = PipelineResource(activities=[Wait_activity]) p = adf_client.pipelines.create_or_update(rg_name, df_name, p_name, p_obj) This is the pipeline before running the code : After running the code: Expected : 回答1: Researched the

问题 I'm actually some automation for my ADF. As a part of that, I'm trying to delete all the ADF V2 pipelines. The problem is my pipelines having many references with different pipelines itself. $ADFPipeline = Get-AzDataFactoryV2Pipeline -DataFactoryName $(datafactory-name) -ResourceGroupName $(rg) $ADFPipeline | ForEach-Object { Remove-AzDataFactoryV2Pipeline -ResourceGroupName $(rg) -DataFactoryName $(datafactory-name) -Name $_.name -Force } And most of the time I get the error like The

问题 I'm actually some automation for my ADF. As a part of that, I'm trying to delete all the ADF V2 pipelines. The problem is my pipelines having many references with different pipelines itself. $ADFPipeline = Get-AzDataFactoryV2Pipeline -DataFactoryName $(datafactory-name) -ResourceGroupName $(rg) $ADFPipeline | ForEach-Object { Remove-AzDataFactoryV2Pipeline -ResourceGroupName $(rg) -DataFactoryName $(datafactory-name) -Name $_.name -Force } And most of the time I get the error like The

问题 I am trying to setup an Azure Data Factory in a CI/CD setup. I followed Microsoft's best practices (https://docs.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment). I have 4 environments (dev, test, UAT, PRD) In short what I have done: Create an Azure data factory and link it to my Azure DevOps repo on DEV environment Create an Azure data factory on the other environments (test, UAT and PRD), but do NOT link it to DevOps. Instead, pipelines are released on these data

问题 I am trying to setup an Azure Data Factory in a CI/CD setup. I followed Microsoft's best practices (https://docs.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment). I have 4 environments (dev, test, UAT, PRD) In short what I have done: Create an Azure data factory and link it to my Azure DevOps repo on DEV environment Create an Azure data factory on the other environments (test, UAT and PRD), but do NOT link it to DevOps. Instead, pipelines are released on these data

问题 I am working on creating an entire CI/CD pipeline for my Data factory V1 project I am using VSTS for the implementation. I am able to carry out most of my task over VSTS which are required for deployment, However, I am not able to determine if it is possible to completely implement continuous deployment over my project. I have one common solution file responsible to hold 4 different data factory project & each project holds 4 data flow pipelines each. The issue is not every time is the case