Count rows in massive .csv file

1
votes

dumping a Postgres table out by sections is yielding sections that are 30GB+ in size. The files are landing on a windows 2008 server. I'm trying to count the rows in the csv to ensure I have a row count that I expect (22,725,303 to be exact). I can count the rows in the section that I expect to dump - but I am not sure if I'm getting them all.

It's a 190M row table so sections of table is the way to go.

so how can I count the rows so I know I've got the full section?

1
postgresqlcsvrowcount
Copy the .csv file(s) to a unix machine and run wc -l thefile.csv on them? - wildplasser
Yeah, I'm not keen to move 300GB around the network. That's onerous. - Matt Coblentz
Maybe cygwin contains the file-utilities? [alternatively, you could boot from an ubuntu-usb-stick, attempt to mount your(ntfs?) disk, and run wc -l] ... Or you could write a small program that just counts the '\n' s. - wildplasser

1 Answers

0
votes

In a PL/pgSQL function, you can get the count of rows processed by the last command - since Postgres 9.3 including COPY - with:

GET DIAGNOSTICS x = ROW_COUNT;