python pickler - recursion depth exceeded

Question

I'm trying to pickle instance of my cellular automata class, but I get this error:

RuntimeError: maximum recursion depth exceeded while calling a Python object

My cellular automata consist from list of cells (and bunch of other things) where each cell has pointer to it's neighbours. In this particular CA, there is 256 cells. Now, I know that pickler should be able to recognise already pickled objects.

From docs:
*The pickle module keeps track of the objects it has already serialized, so that later references to the same object won’t be serialized again.

So I don't really know, why I exceeding max recursion depth.

I think that maybe pickler does depth-first pickling, so that it first follow pointers, exceed recursion stack and then raise exception. I know I can extend maximum recursion depth with sys.setrecursionlimit(), but I don't consider that good nor scalable solution.

First question: Does pickler depth-first pickling?
Second question: Any idea how to prevent this exception?

Yes, pickle goes depth-first. Unfortunately I don't think there is a around this. Try cPickle but it'll probably give the same error. — Elmar Peise
@AswinMurugesh actually I have too much code to post. However here are some key files if you are interested: neighbourhood cell pickling Exp: cPickle is the same, I tried it before. I was afraid of this answer — sjudǝʊ
@ExP do you have some more specific info / resources about how pickle works internally? Maybe creating subclass of pickler? There's always way. I just need to take a closer look :) — sjudǝʊ
@sjudǝʊ If you look at the output of pickle given dummy data, you will see that it gos depth first. Since I guess that the format does not allow forward references, I'm afraid you won't be able to modify or create subclass of it in order to solve your problem... — Elmar Peise

sjudǝʊ sjudǝʊ · Accepted Answer · 2013-04-16T18:07:19

So, as @ExP said, pickler does depth-first pickling which causes recursion exceeded exception. Anyway, I found solution to this issue here bugs.python.org. That means for python 3.1 pickler works even on recursive data such as graphs for example.

There is also little less elegant solution which takes a lot more time to pickle some recursive data, but it's simple (just a matter of few lines of code). Link here.

As it seems, it's probably time to start slowly moving towards the python3. Hope that someone find this answer usefull.

python pickler - recursion depth exceeded

1 Answers