Connect to external services inside Visual Studio Online build/test task

Question

We are running our build and test process on TFS Online. This works fine, until we try to connect to an external service. In our case a SQL Database.
The discussion about what we should mock and not is not really helpful in this case, as currently we need to do this.

We also tried just a simple ping, but not even that is getting out:

Test-Connection "172.217.18.100" #resolves to www.google.com

Testing connection to computer '172.217.18.100' failed: Error due to lack of resources

So we have the impression that most outside IP's/Ports/etc. could be locked? Is there a way to open this up? If yes, how?

I can't imagine that we are the first ones to try something like that? Downloading something from a website, making a REST all, etc? Should be possible somehow, no?

Update 1:
We had a bit a more detailed question about this issue here, but figured it was a more generic problem.

The error message when we connect to Azure SQL is

System.Data.SqlClient.SqlException: 
A network-related or instance-specific error occurred while establishing a    connection to SQL Server. 
The server was not found or was not accessible. 
Verify that the instance name is correct and that SQL Server is configured to allow remote connections. 
(provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server)`

But the Named Pipes Provider part is missleading. You also get this message locally if you try to connect to an IP that does not exist.

We access Azure SQL from our C# NUnit tests.

Update 2:
We tried out the idea from @starain-MSFT and installed the Azure SQL Execute Query step/task. Installation works fine, but it seems to miss a component. See picture below.

No agent could be found with the following capabilities: azureps, sqlps, npm, node.js, msbuild, visualstudio, vstest

Based on installed apps list I assume it's azureps.

Solution (partly):
Ok, so we went down the wrong route all the way. The problem is not the firewall (or any firewalls for that matter). The problem was that our app.config file didn't had the right settings in it.
We had the same setup for our App.config files inside the unit tests as for our Web.config files. Each had a App.Debug.config and a App.Release.config file attached. While this worked fine for the web applications, this obviously didn't work for our unit tests.
We are still looking for a good solution here. I found this solution on how to add a transform task inside Visual Studio, but this is not exactly what we are searching, because we don't need the transform locally, but only in Visual Studio Teams.

app.config Transform inside Visual Studio Teams
So, I think we finally got it. With ConfigTransform, we can now transform our app.config files during the build process.

Answer 1

Windows Hosted build agents don't block 1433 outbound.

1. If you want to connect to SQL Azure through hosted build agents ensure that you enabled in your SQL Azure firewall settings "Allow access to Azure services". You don't need to run a script manually.

SQL Azure Firewall settings

2. Ensure you are using the right connection string during unit testing. E.g. in MSTest you need to add your connection string into App.config of you UnitTest project.

<connectionStrings>
  <add name ="TestContext" providerName="System.Data.SqlClient" 
  connectionString="Server=tcp:[ServerName].database.windows.net,1433;Initial Catalog=[DB Name];Persist Security Info=False;User ID=[User];Password=[Password];MultipleActiveResultSets=False;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;"/>
</connectionStrings>

That's it. I just did a quick test with EF, SQL Azure & VSTS Hosted Agent and it worked.

Answer 2

Using Hosted Agents, the SQL Server need to be accessible from internet in order to connect to your SQL Server from Hosted Agents.

The way to deal with this issue:

1. As Giulio said that set up an on premise build agent, then you just need to make sure the SQL Server instance can be accessible from that build agent (can be intranet).
2. Apply a SQL Server on internet, such as Azure SQL Server that can be accessible from internet.
3. Configure your SQL Server and network to let your SQL Server can be accessible from internet.

BTW, regarding your simple ping test, that IP address is used for its web site and the port is 80, you can access other resource with that IP. You can open another port on your server and access resource by IP with port.

Update 1:

Refer to this way to add Azure SQL Server Firewall Rule:

1. Check Allow Scripts to Access OAuth Token option (Options of build definition)
2. Add Azure PowerShell build step before test step (Arguments: -RestAddress https://[your vsts account].vsdtl.visualstudio.com/DefaultCollection/_apis/vslabs/ipaddress -Token $(System.AccessToken) -RG [resource group] -Server [server name(without .database.windows.net)]

Script:

param (
    [string]$RestAddress,
    [string]$Token,
    [string]$RG,
    [string]$Server
    )
$basicAuth = ("{0}:{1}" -f 'test',$Token)
$basicAuth = [System.Text.Encoding]::UTF8.GetBytes($basicAuth)
$basicAuth = [System.Convert]::ToBase64String($basicAuth)
$headers = @{Authorization=("Basic {0}" -f $basicAuth)}
$result = Invoke-RestMethod -Uri $RestAddress -headers $headers -Method Get
Write-Host $result.value
New-AzureRmSqlServerFirewallRule -ResourceGroupName $RG -ServerName $Server -FirewallRuleName "UnitTestRule" -StartIpAddress "$($result.value)" -EndIpAddress "$($result.value)"

BTW, you can refer to that script to remove the firewall rule after test.

Update 2:

The SQL ConnectionString like this:

Server=tcp:[server name].database.windows.net,1433;Initial Catalog=sqlstarain1;Persist Security Info=False;User ID=[user name];Password=[password];MultipleActiveResultSets=False;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;

Answer 3

I assume that your are using the Hosted Agents, which means that the machine is a shared resource between many VSTS accounts (tenants) and managed (and locked) down by Microsoft.

You can easily install an agent on your own virtual machine and run the build there. The VM can be in the cloud or on premise, your choice. You trade simplicity and cheapness for full control.

Update: Hosted Agents allows HTTP(S) calls which cover a lot of grounds. While useful I do not think it solves the original question to connect to a SQL database.