Powershell script been running for days when doing comparison

Question

I got a powershell query, it works fine for smaller amount of data but i am trying to run my CSV against a folder which has multiple folders and files within. Folder size is nearly 800GB and 180 folders within.

I want to see if the file exists in the folder, I can manually search the files within Windows and does not take to long to return a result but my CSV has 3000 rows and i do not wish to do this for 3000 rows. My script works fine for a smaller amount of data.

The script has been running for 6 days and it has not generated a file with data as of yet. it is 0KB and I am running it via task scheduler.

Script is below.

$myFolder = Get-ChildItem 'C:\Test\TestData' -Recurse -ErrorAction 
SilentlyContinue -Force
$myCSV = Import-Csv -Path 'C:\Test\differences.csv' | % {$_.'name' -replace "\\", ""}
$compare = Compare-Object -ReferenceObject $myCSV -DifferenceObject $myFolder

Write-Output "`n_____MISSING FILES_____`n" 
$compare

Write-Output "`n_____MISSING FILES DETAILS____`n"
foreach($y in $compare){
if($y.SideIndicator -eq "<="){
write-output "$($y.InputObject) Is present in the CSV but not in Missing folder." 

}
}

I then created another script which runs the above script and contains an out file command and runs with Task scheduler. C:\test\test.ps1 | Out-File 'C:\test\Results.csv'

is there a better way of doing this?

Thanks

Would compare-object -syncwindow 1 do the job? Otherwise it compares every line with every line. — js2010
An example line from your C:\Test\differences.csv would probably help as well. There may be a faster route with a different approach for what you're trying to do than Compare-Object, which is possibly comparing file sizes, last write times, etc, and all you care about is presumably file names. — mclayton
@js2010 i never seen that command before, what does it do -syncwindow 1? — newbie9803
It will only compare 1 line above or below within each file. Otherwise it compares all lines with all lines and the order doesn't matter. The default syncwindow is very slow for large files. You also need to specify what properties to compare. — js2010
@newbie9803 - the column headers and an example line would be useful. — mclayton

Mathias R. Jessen Mathias R. Jessen · Accepted Answer · 2021-05-06T15:04:35

is there a better way of doing this?

Yes!

Add each file name on disk to a HashSet[string]
- the HashSet type is SUPER FAST at determining whether it contains a specific value or not, much faster than Compare-Object
Loop over your CSV records, check if each file name exists in the set from step 1

# 1. Build our file name index using a HashSet
$fileNames = [System.Collections.Generic.HashSet[string]]::new()
Get-ChildItem 'C:\Test\TestData' -Recurse -ErrorAction 
SilentlyContinue -Force |ForEach-Object {
  [void]$fileNames.Add($_.Name)
}

# 2. Check each CSV record against the file name index
Import-Csv -Path 'C:\Test\differences.csv' |ForEach-Object {
  $referenceName = $_.name -replace '\\'
  if(-not $fileNames.Contains($referenceName)){
    "${referenceName} is present in CSV but not on disk"
  }
}

Another option is to use the hash set from step 1 in a Where-Object filter:

$csvRecordsMissingFromDisk = Import-Csv -Path 'C:\Test\differences.csv' |Where-Object { -not $fileNames.Contains($_) }

Powershell script been running for days when doing comparison

1 Answers