0
votes

What is the fastest way to Temporary store huge amount of data?

I'm writing a program that would be reading data from the database in chunks and I need to store it temporarily in some file. After that, I would be passing it with REST API.

Right now I'm considering StringIO vs TemporaryFile. Looks like first saves data in memory and the second one on disk - but the question is which one is fastest for the big volumes of data?

Memory is always going to be faster if you have enough available. But you also might want to check whether you really need to store a large amount temporarily - can you change your workflow to iterate over your data, reading in parts of your data only at the time when you are ready to process them, rather than reading it all in at the start? - alani
you can use glob - Ahx
You could easily answer the question by doing a simple test. Code for it would be shorter than the question. - Klaus D.
@alaniwi I totally understand, but I can't solve this case stackoverflow.com/questions/62711507/… in a more simple way, so I decided to store it temporarily. - venv
@VenusV Okay, fair enough. It's always worth asking. I've seen people's code that loads huge multi-dimensional arrays into memory when the calculation isn't doing anything that couldn't be done inside a loop over the largest and slowest-varying dimension... - alani