4
votes

Run this:

for i in range(1000000000):
    a = []

It looks like the list objects being created never get marked for garbage collection. From a memory profiler, it looks like the interpreter's stack frame is holding onto all the list objects, so GC can never do anything about it.

Is this by design?

EDIT:

Here is a better example of the problem. Run the code below with a memory profiler:

a = [b for b in range(1000000)]
a = [b for b in range(1000000)]
a = [b for b in range(1000000)]
a = [b for b in range(1000000)]
a = [b for b in range(1000000)]
a = [b for b in range(1000000)]
a = [b for b in range(1000000)]

You will see that the memory allocated during the list comprehensions never gets garbage collected. This is because all the objects created are being referenced by an InterpreterFrame object in the DLR.

Now run this:

def get():
    return [b for b in range(1000000)]

a = get()
a = get()
a = get()
a = get()
a = get()
a = get()
a = get()

Under a profiler, you can see that the memory here does get garbage collected as it should. I am guessing this works because the InterpreterFrame of the function is cleared when the function exits.

So, is this a bug? This seems that it will lead to some pretty bad memory leaks when within frames (contexts?) of an IronPython script.

1
Do you explicitely run gc.collect()? - doublep
What about using xrange? range will create a huge list which won't go away until the loop is done anyway. - Dino Viehland
CPython 2.6.5 does that too, it raises: Traceback (most recent call last): File "<pyshell#0>", line 1, in <module> for i in range(1000000000): MemoryError. xrange works, but it takes forever for both CPython and IronPython for me, so I just ctrl-C. - jcao219
The problem is not in the range function...see my edit - Mike Gates

1 Answers

1
votes

Try setting "LightweightScopes" when you are constructing IronPython engine. This solved a lot of garbage collection problems for me.

var engineOptions = new Dictionary<string, object> { ["LightweightScopes"] = true };
var scriptEngine = Python.CreateEngine(engineOptions);