Running multiple scriptblocks at the same time with Start-Job (instead of looping)

Question

Hi all!

I\'ve been looking for a way to make my script more efficient and I\'ve come to the conclusion (with help from the nice people here on Stack

Answer 1

Since your loop only needs to work with a string it's easy to turn it into a concurrent script.

Below is an example of making making your loop use background jobs to speed up processing.

The code will loop through the array and spin up background jobs to run the code in the script block $sb. The $maxJobs variable controls how many jobs run at once and the $chunkSize variable controls how many servers each background job will process.

Add the rest of your processing in the script block adding whatever other properties you want to return to the PsObject.

$sb = {
    $serverInfos = @()
    $args | % {
        $IPAddress = [Net.Dns]::GetHostAddresses($_) | select -expand IPAddressToString
        # More processing here... 
        $serverInfos += New-Object -TypeName PsObject -Property @{ IPAddress = $IPAddress }
    }
    return $serverInfos
}

[string[]] $servers = Get-QADComputer -sizelimit 500 -WarningAction SilentlyContinue -OSName *server*,*hyper* | Select -Expand Name

$maxJobs = 10 # Max concurrent running jobs.
$chunkSize = 5 # Number of servers to process in a job.
$jobs = @()

# Process server list.
for ($i = 0 ; $i -le $servers.Count ; $i+=($chunkSize)) {
    if ($servers.Count - $i -le $chunkSize) 
        { $c = $servers.Count - $i } else { $c = $chunkSize }
    $c-- # Array is 0 indexed.

    # Spin up job.
    $jobs += Start-Job -ScriptBlock $sb -ArgumentList ( $servers[($i)..($i+$c)] ) 
    $running = @($jobs | ? {$_.State -eq 'Running'})

    # Throttle jobs.
    while ($running.Count -ge $maxJobs) {
        $finished = Wait-Job -Job $jobs -Any
        $running = @($jobs | ? {$_.State -eq 'Running'})
    }
}

# Wait for remaining.
Wait-Job -Job $jobs > $null

$jobs | Receive-Job | Select IPAddress

Here is the version that processes a single server per job:

$servers = Get-QADComputer -WarningAction SilentlyContinue -OSName *server*,*hyper*

# Create list
$serverlist = @()

$sb = {
    param ([string] $ServerName)
    try {
        # Fetch IP
        $ipaddress = [System.Net.Dns]::GetHostAddresses($ServerName)| select-object IPAddressToString -expandproperty IPAddressToString

        # Gather OSName through WMI
        $OSName = (Get-WmiObject Win32_OperatingSystem -ComputerName $ServerName ).caption

        # Ping the server
        if (Test-Connection -ComputerName $ServerName -count 1 -Quiet ) {
            $reachable = "Yes"
        }

        # Save info about server
        $serverInfo = New-Object -TypeName PSObject -Property @{
            SystemName = ($ServerName).ToLower()
            IPAddress = $IPAddress
            OSName = $OSName
        }
        return $serverInfo
    } catch {
        throw 'Failed to process server named {0}. The error was "{1}".' -f $ServerName, $_
    }
}

# Loop servers
$max = 5
$jobs = @()
foreach($server in $servers) {
    $jobs += Start-Job -ScriptBlock $sb -ArgumentList $server.Name
    $running = @($jobs | ? {$_.State -eq 'Running'})

    # Throttle jobs.
    while ($running.Count -ge $max) {
        $finished = Wait-Job -Job $jobs -Any
        $running = @($jobs | ? {$_.State -eq 'Running'})
    }
}

# Wait for remaining.
Wait-Job -Job $jobs > $null

# Check for failed jobs.
$failed = @($jobs | ? {$_.State -eq 'Failed'})
if ($failed.Count -gt 0) {
    $failed | % {
        $_.ChildJobs[0].JobStateInfo.Reason.Message
    }
}

# Collect job data.
$jobs | % {
    $serverlist += $_ | Receive-Job | Select-Object SystemName,IPAddress,OSName
}