python - Why pickle eat memory?

Question

I trying to deal with writing huge amount of pickled data to disk by small pieces. Here is the example code:

from cPickle import *
from gc import collect

PATH = r'd:\test.dat'
@profile
def func(item):
    for e in item:
        f = open(PATH, 'a', 0)
        f.write(dumps(e))
        f.flush()
        f.close()
        del f
        collect()

if __name__ == '__main__':
    k = [x for x in xrange(9999)]
    func(k)

open() and close() placed inside loop to exclude possible causes of accumulation of data in memory.

To illustrate problem I attach results of memory profiling gained with Python 3d party module memory_profiler:

   Line #    Mem usage  Increment   Line Contents
==============================================
    14                           @profile
    15      9.02 MB    0.00 MB   def func(item):
    16      9.02 MB    0.00 MB       path= r'd:\test.dat'
    17
    18     10.88 MB    1.86 MB       for e in item:
    19     10.88 MB    0.00 MB           f = open(path, 'a', 0)
    20     10.88 MB    0.00 MB           f.write(dumps(e))
    21     10.88 MB    0.00 MB           f.flush()
    22     10.88 MB    0.00 MB           f.close()
    23     10.88 MB    0.00 MB           del f
    24                                   collect()

During execution of the loop strange memory usage growth occurs. How it can be eliminated? Any thoughts?

When amount of input data increases volume of this additional data can grow to size much greater then input (upd: in real task i get 300+Mb)

And more wide question - which ways exist to properly work with big amounts of IO data in Python?

upd: I rewrote the code leaving only the loop body to see when growth happens specifically, and here the results:

Line #    Mem usage  Increment   Line Contents
==============================================
    14                           @profile
    15      9.00 MB    0.00 MB   def func(item):
    16      9.00 MB    0.00 MB       path= r'd:\test.dat'
    17
    18                               #for e in item:
    19      9.02 MB    0.02 MB       f = open(path, 'a', 0)
    20      9.23 MB    0.21 MB       d = dumps(item)
    21      9.23 MB    0.00 MB       f.write(d)
    22      9.23 MB    0.00 MB       f.flush()
    23      9.23 MB    0.00 MB       f.close()
    24      9.23 MB    0.00 MB       del f
    25      9.23 MB    0.00 MB       collect()

It seems like dumps() eats memory. (While I actually thought it will be write())

First, you're only at 11MB. Are you sure there's a real problem? Have you actually tried it with large amounts of data to see if it increases linearly to some scary level? Second, the increment happens on the for loop (so presumably inside item.__next__), not the dumps line. (And if you do think it's the pickling, why haven't you tried splitting dumps and write into separate steps?) — abarnert
Also, memory_profiler says it "gets the memory consumption by querying the operating system kernel about the amount of memory the current process has allocated, which might be slightly different from the ammount of memory that is actually used by the Python interpreter". In fact, it may be way, way different! Just because Python calls free doesn't necessarily mean the platform's allocator releases it all immediately to the OS—in fact, it's perfectly reasonable for it to hold onto the page mappings and never release them. — abarnert
For your "wide question": It depends on how big you mean by big. But the two basic strategies are: don't use that much (e.g., use a numpy array of ints instead of a name list of lists of Python objects), or use a database (anydbm or sqlite3) instead of building a giant in-memory store and persisting it to disk en masse. — abarnert
Check out streaming-pickle which supposedly would use a lot less memory for what you're doing. — martineau
@GillBates: Have you tested, .e.g, just storing your data in a shelve to see if it actually does use memory this way, instead of assuming it must because it uses pickle? Also, does your peak data usage actually overrun your bounds (or, if you're on 64-bit, throw you into swap thrash hell)? There are some use cases in Python that seem to be linear in space, but are actually just linear up to some constant limit after which they flatten out (by reusing that same storage). Especially if you're on a platform that doesn't usually release memory to the kernel and you're measuring from outside. — abarnert

AMairesse AMairesse · Accepted Answer · 2016-08-16T09:38:39

Pickle consume a lot of RAM, see explanations here : http://www.shocksolution.com/2010/01/storing-large-numpy-arrays-on-disk-python-pickle-vs-hdf5adsf/

Why does Pickle consume so much more memory? The reason is that HDF is a binary data pipe, while Pickle is an object serialization protocol. Pickle actually consists of a simple virtual machine (VM) that translates an object into a series of opcodes and writes them to disk. To unpickle something, the VM reads and interprets the opcodes and reconstructs an object. The downside of this approach is that the VM has to construct a complete copy of the object in memory before it writes it to disk.

Pickle is great for small use cases or testing because in most case the memory consumption doesn't matter a lot.

For intensive work where you have to dump and load a lot of files and/or big files you should consider using another way to store your data (ex.: hdf, wrote your own serialize/deserialize methods for your object, ...)

python - Why pickle eat memory?

1 Answers