Writing 100,000 lines in a csv in fastest way possible

0
votes

I m trying to create a zip of 100 csv. I need to write 100,000 lines in a single csv in fastest way possible. I am using openCSV, ZipEntry method to write to csv.

Some Code used: 

ZipEntry zipentry = new ZipEntry(filename);
zos.putNextEntry(entry);
CSVWriter writer = new CsvWriter(new OutputStreamWriter(zos));
writer.writeNext(entries); //entries is single line of csv

Currently its taking 1.5 secs to write a single csv and overall its taking around 120-140secs to create complete zip.

I have debugged the code, and observed that the other computations in the code is not taking time, but the write operation does takes time.

I have tried creating list of 100,000 lines and then writing one one file at a time instead if direct streaming. But still it takes same time.

Please suggest fastest method which takes less time. ;-(

2
javacsvopencsvfast-csv
A hard disc is inherently a slow device. If you really need speedy IO consider a solid state drive (and yes, I mean in a production system). – markspace
But i want to do it from java perspective. Any suggestions for that? – NobesInd
Your can run each write asynchronously to speed up the process. Try ten at a time and you should be done in 15 seconds. – Nikhil
I can't do async, since the data which is getting write is computated first and they r dependent on previous data. – NobesInd

2 Answers

2
votes

BufferedOutputStream improves IO performance by a large margin:

see Javadoc: BufferedOutputStream

By setting up such an output stream, an application can write bytes to the underlying output stream without necessarily causing a call to the underlying system for each byte written.

OutputStream out = new BufferedOutputStream(zos, 1<<16);
CSVWriter writer = new CsvWriter(new OutputStreamWriter(out);
1
votes

Try using less aggressive compression level for your ZipOutputStream such as 3:

public void setLevel(int level)