Not all of my PowerShell v2 jobs get run

Question

I'm trying to use jobs to have simultaneous instances of PowerShell run, so that my script can finish faster. The scripts I'm calling are all independent, no need for mutual exclusion etc. Problem is that it will call five out of 50 scripts and then it will stop. The rest of the scripts never seem to run. Something wrong with wait-job? Also for some reason the five scripts that actually run seem to be executed twice so I get double output...

$Invocation = (Get-Variable MyInvocation -Scope 1).Value
$ScriptDir = Split-Path $Invocation.MyCommand
$ScriptDir -match '(?<content>.*\d)' | out-null
$MainDir = $matches['content']
$ScriptName = $MyInvocation.MyCommand.Name
$Items = Get-ChildItem -Path $MainDir | Where-Object {$_.mode -match "d"}

$jobMax = 4
$jobs = @()

$jobWork = {
    param ($MyInput,$dir)
    $command = "$dir\" + $MyInput.name + "\" + $MyInput.name + ".ps1"
    Start-Process powershell.exe -argumentlist $command -WindowStyle Hidden #-wait
}

foreach ($Item in $Items) {
    if ($jobs.Count -le $jobMax) {
        $jobs += Start-Job -ScriptBlock $jobWork -ArgumentList $Item,$MainDir
    } else {
        $jobs | Wait-Job -Any
    }
}
$jobs | Wait-Job

Edit: Also, since I'm using start-process powershell all the scripts run simultaneously(unless I enable -wait), without the need for start-job. I wanted to use jobs so that I can throttle. Maybe it's wrong logic that I have a job start a new instance of powershell?

Edit2: As I thought it was wrong to use start-job to start a new instance of poweshell. That way the job was done after it opened the new powershell instance, the actual script contents had nothing to do with the job's start and finish. Here is my fixed script :)

$Invocation = (Get-Variable MyInvocation -Scope 1).Value
$ScriptDir = Split-Path $Invocation.MyCommand
$ScriptDir -match '(?<content>.*\d)' | out-null
$MainDir = $matches['content']
$ScriptName = $MyInvocation.MyCommand.Name
$Items = Get-ChildItem -Path $MainDir | Where-Object {$_.mode -match "d"}

$maxJobs = 4
$jobs = @()

foreach ($Item in $Items) {
    $command = "$MainDir\" + $Item.name + "\" + $Item.name + ".ps1"
    $jobs += Start-Job -filepath $command -ArgumentList $MainDir,$Item 
    $running = @($jobs | ? {$_.State -eq 'Running'})

    while ($running.Count -ge $maxJobs) {
        $finished = Wait-Job -Job $jobs -Any
        $running = @($jobs | ? {$_.State -eq 'Running'})
    }      
}
Wait-Job -Job $jobs > $null

I had to edit all my script so that I could pass the arguments as parameters, but now everything works fine. This thread was helpful Running multiple scriptblocks at the same time with Start-Job (instead of looping)

BartekB BartekB · Accepted Answer · 2012-06-16T16:11:50

My guess is (after looking at your code) is that you will get only first 4 jobs started. Jobs (even completed) do not disappear automagically. You need to clean them once they are completed (maybe Receive-Job's first). If you won't do that - PowerShell will spin $jobsMax jobs and once it's done - I would be really surprised to see any other "newcomers".

Add some Receive/Remove job logic and it should work better.

Also: as far as I can tell you are running scripts, so I would suggest using -File parameter on powershell.exe rather than -command.

HTH Bartek

Not all of my PowerShell v2 jobs get run

1 Answers