问题
I am trying to setup an Azure Data Factory in a CI/CD setup. I followed Microsoft's best practices (https://docs.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment).
I have 4 environments (dev, test, UAT, PRD)
In short what I have done:
Create an Azure data factory and link it to my Azure DevOps repo on DEV environment
Create an Azure data factory on the other environments (test, UAT and PRD), but do NOT link it to DevOps. Instead, pipelines are released on these data factories using ARM templates and release pipelines in Azure DevOps.
I have parameterized all necessary parts to be able to overwrite the settings in each of my environments.
At this moment, I am able to succesfully deploy to my other environments, however, the linkedservice to my database on azure is not working. I have parameterized everything, like Microsoft suggests, but when I export my linkedservice to an ARM template, it uses a connection string instead of my parameterized settings.
Below is a picture of my settings in the Azure Data Factory portal:
When I try to export this to ARM templates, I get the following:
{
"name": "[concat(parameters('factoryName'), '/database')]",
"type": "Microsoft.DataFactory/factories/linkedServices",
"apiVersion": "2018-06-01",
"properties": {
"parameters": {
"sqlServerUrl": {
"type": "string"
},
"databaseName": {
"type": "string"
},
"sqlPwd": {
"type": "string"
},
"sqlAdminUsername": {
"type": "string"
}
},
"annotations": [],
"type": "AzureSqlDatabase",
"typeProperties": {
"connectionString": {
"type": "SecureString",
"value": "[parameters('database_connectionString')]"
}
}
},
"dependsOn": []
},
The problem with this ARM templates is that it does not use the parameters to create the connection string, but it uses the connection string parameter database_connectionString (the connection string is by default always parameterized by Azure, so I cannot remove this parameter).
When the release pipeline uses this template, the connectionstring is not filled in (only the parameters are filled in) and hence, the connection to the database fails. How should you setup the connection so that you can automatically (without human interaction) deploy to all environments by only changing the parameters?
I do not want to change the ARM templates coming from Azure Data Factory, because this requires human interaction
回答1:
One of the other answers provided is a valid way using override parameters. This answer will provide a different answer as well as some more context on how to define the SQL connections and how to utilize and implement some of the updates made with Key Vault and Data Factory Integration.
If using an on prem connection to SQL the conenction string will look like:
"Server={serverName};Database={databaseName};User ID={domain}\{userName};Password={password};Integrated Security=True;"
The quotes are required and can be passed in as an override parameter.
If using an Azure Database or even using Key Vault look to add Managed Identity which Data Factory Supports by including this in your ARM template
"identity": {
"type": "SystemAssigned"
}
Once this is added then the Azure SQL Database will need to have the managed identity added. This can be done via a reusable SQL Script like:
DECLARE @rolename AS NVARCHAR(100) = 'DataFactory_User'
DECLARE @username AS NVARCHAR(100) -- This will be the DataFactory name
SET @username = 'DataFacotryName'
if exists(select * from sys.database_principals where name = @username and Type = 'X' or Type='E')
BEGIN
DECLARE @dropUserSql varchar(1000)
SET @dropUserSql='DROP USER [' + @username + ']'
PRINT 'Executing ' + @dropUserSql
EXEC (@dropUserSql)
PRINT 'Completed ' + @dropUserSql
END
DECLARE @createUserSql varchar(1000)
SET @createUserSql='CREATE USER [' + @username + '] FROM EXTERNAL PROVIDER'
PRINT 'Executing ' + @createUserSql
EXEC (@createUserSql)
PRINT 'Completed ' + @createUserSql
I recommend dropping and recreating this user. SQL recognizes the thumbprint of the Managed Identity and everytime the DataFactory is dropped and recreated a new thumbprint is created.
In terms of leveraging Key Vault there is a LinkedService type of Key Vault that relies on the Managed Identity described above to retrieve secrets.
The Key Vault if deployed via ARM will need to have the access policy updated to something similar to this:
"accessPolicies": [
{
"tenantID": "[subscription().tenantId]",
"objectId": "[reference(resourceId('Microsoft.DataFactory/factories/', parameters('DataFactoryName')), '2018-02-01', 'Full').identity.principalId]",
"permissions": {
"secrets": [
"get"
],
"keys": [
"get"
],
"certificates": [
"import"
]
},
"dependsOn": [
"[resourceId('Microsoft.DataFactory/factories/', parameters('DataFactoryName'))]"
]
}
]
This snippet assumes the Key Vault and Data Factory are in the same ARM template. If they are not the access policy can still be accomplished via ARM by obtaining the ObjectId of the Data Factory defined Managed Identity and passing it in as the ObjectId and removing the dependsOn statement.
回答2:
I do this by using the Azure resource group deployment task in the Release. One of the options is Override template parameters which will allow you to do exactly what you need. You can create a space separated list of parameters to override your ARM Template and pass in a Variable
In your case it would be
-database_connectionstring $(VariableHere)
I would store the connection string in Azure KeyVault and link it to a Secure Variable Group. You can also just hit the padlock on a standard variable to secure it.
Then bind your custom variable to each Stage in your Release
Task
Override
来源：https://stackoverflow.com/questions/57753824/parameterize-connections-in-azure-data-factory-arm-templates