PowerShell how to Split a text file based into multiple based on a change in value in a string

Question

I am new to Powershell and I need help with a requirement which can be give results (executed) very fast using PowerShell.

I have a text file that has data which contains about 500000 records and about 100 delimited (|) fields. The first field contains date in YYYYMMDD format. I need to split the large file each time there is a change in this value. For example if the file has about 100 different dates the file should be split in to 100 files.

I am giving a sample set of data for reference with only 3 fields. The name of the file in test1.txt. The resultant files should be named like test1_00001.txt, test1_00002.txt,......,test1_00100.txt.

Suppose input file test.txt contains below records:

20190227|00001|VALUE1  
20190227|00001|VALUE1  
20190227|00001|VALUE2  
20190227|00002|VALUE3  
20190228|00005|VALUE3  
20190228|00001|VALUE2  
20190228|00002|VALUE1  
20190301|00001|VALUE1  
20190301|00300|VALUE7  
20190301|00004|VALUE7  
20190301|00004|VALUE1  
20190301|00002|VALUE5

Resultant file test_00001.txt should contain below records:

20190227|00001|VALUE1  
20190227|00001|VALUE1  
20190227|00001|VALUE2  
20190227|00002|VALUE3

Resultant file test_00002.txt should contain below records:

20190228|00005|VALUE3  
20190228|00001|VALUE2  
20190228|00002|VALUE1

Resultant file test_00002.txt should contain below records:

20190301|00001|VALUE1  
20190301|00300|VALUE7  
20190301|00004|VALUE7  
20190301|00004|VALUE1  
20190301|00002|VALUE5

Possible duplicate of How to split one big text file into multiple files in powershell — user6811411

Unknown Unknown · Accepted Answer · 2019-04-28T12:01:15

Why not use the date as part of the new file name?

Get-Content .\test1.txt | %{Add-Content -path "test$($_.split('|')[0]).txt" -Value $_}

You may experiment with Get-Content -ReadCount parameter to speed up.

PowerShell how to Split a text file based into multiple based on a change in value in a string

1 Answers