How can I call many URLs from a list asynchronously

Question

I have a few hundred thousand URLs that I need to call. These are calls to an application server which will process them and write a status code to a table. I do not need to

Answer 1

With Jobs you incur a large amount of overhead, because each new Job spawns a new process.

Use Runspaces instead!

$maxConcurrentJobs = 10
$content = Get-Content -Path "C:\Temp\urls.txt"

# Create a runspace pool where $maxConcurrentJobs is the 
# maximum number of runspaces allowed to run concurrently    
$Runspace = [runspacefactory]::CreateRunspacePool(1,$maxConcurrentJobs)

# Open the runspace pool (very important)
$Runspace.Open()

foreach ($url in $content) {
    # Create a new PowerShell instance and tell it to execute in our runspace pool
    $ps = [powershell]::Create()
    $ps.RunspacePool = $Runspace

    # Attach some code to it
    [void]$ps.AddCommand("Invoke-WebRequest").AddParameter("UseBasicParsing",$true).AddParameter("Uri",$url)

    # Begin execution asynchronously (returns immediately)
    [void]$ps.BeginInvoke()

    # Give feedback on how far we are
    Write-Host ("Initiated request for {0}" -f $url)
}

As noted in the linked ServerFault post, you can also use a more generic solution, like Invoke-Parallel, which basically does the above