Memory Leak in Threaded COM Object with Python

Question

I am creating a COM client within a thread and performing several operations with this client. Each thread is spawned from a server that uses Python's socketserver module which has built-in threading support.

When I am loading and using this COM object there is an expected spike in memory usage by python.exe. With 10 concurrent threads there is a peak in memory usage of about 500Mb. However when the operations are finished and the COM object is apparently released, there are 50Mb of additional memory used by the process than before. If I then spawn 10 additional threads using the same server there are 13Mb additional used by python.exe after those COM objects are closed. Eventually every 10 additional concurrent threads adds approximately 6Mb after they are done. When I end the entire python.exe process, all the memory is released.

I have simplified the following code to mimic how the socketserver uses threadding and the problem is the exact same.

import win32com.client
import threading
import pythoncom

def CreateTom():
    pythoncom.CoInitialize()
    tom = win32com.client.Dispatch("TOM.Document")
    tom.Dataset.Load("FileName")
    tom.Clear()
    pythoncom.CoUninitialize()

for i in range(50):
    t = threading.Thread(target = CreateTom)
    t.daemon = False
    t.start()

I understand that I will unlikely get any support here around the specific COM library (it is an IBM product used in Market Research known as the TablesObjectModel). However I want to know if there is anything, ANYTHING, additional that I can do to release this memory. I have read about Apartments in COM but it sounds like pythoncom.CoInitialize should take care of this for me. Any help would be appreciated.

This might not be entirely relevant, but try use a thread pool instead of spawning new thread without defining any constraints like the example you posted. — woozyking
Is the TOM.Document in-process or out-of-process? Are you sure the memory is not released after "some time"? This can happen for example with COM Objects written in technologies with a garbage collector (such as .NET, though I doubt TOM.Document is written in .NET) — Simon Mourier
TOM.Document is in-process. It does not fork another process outside python.exe — Michael David Watson
Creating a separate subprocess for the COM object did not work out. Because of the amount of the cross-process communication, I could not get this to work with pipes on Windows. — Michael David Watson

Michael David Watson Michael David Watson · Accepted Answer · 2013-05-17T20:08:37

As it turns out this increase in Memory was in fact due to the COM object written in .NET and had nothing to do with threading. Here is a detailed description of Task Manager giving misleading information about memory usage for .NET apps. To resolve this issue I added the following to my code and I am all set. Hopefully someone else reads this response before they start tearing their hair out trying to find a memory leak in their code.

from win32process import SetProcessWorkingSetSize
from win32api import GetCurrentProcessId, OpenProcess
from win32con import PROCESS_ALL_ACCESS

import win32com.client
import threading
import pythoncom

def CreateTom():
    pythoncom.CoInitialize()
    tom = win32com.client.Dispatch("TOM.Document")
    tom.Dataset.Load("FileName")
    tom.Clear()
    pythoncom.CoUninitialize()
    SetProcessWorkingSetSize(handle,-1,-1) #Releases memory after every use

pid = GetCurrentProcessId()
handle = OpenProcess(PROCESS_ALL_ACCESS, True, pid)

for i in range(50):
    t = threading.Thread(target = CreateTom)
    t.daemon = False
    t.start()

Memory Leak in Threaded COM Object with Python

3 Answers