I am integrating IronPython scripts to run under a C# engine. The C# engine builds a dictionary of 'ScriptValue' objects and passes it to the IronPython Script which then uses the objects to do calculations. The 'ScriptValue' object is in a separate class library and implements 'MarshalByRefObject' and is a simple .net object (stores double and bool values only). The script run happens frequently.
First Attempt: I instantiated the IronPython engine and ran the scripts. As the runs progress I could see that memory usage was increasing at a fast rate. Eventually after a day or running the application crashed with an out of memory exception. I tried both keeping an instance of the IronPythonEngine alive and restarting a new instance on each run. I also tried shutting down the IronPython Engine but memory would increase consistently.
Second Attempt: After researching this a lot, suggestions came up to try running the engine in a separate AppDomain and unloading the AppDomain once you are done running the scripts. I then implemented this and create a new AppDomain and unload it once a run has completed. This appears to help to a certain degree but the memory leak persists albeit it creeps up at a slower rate.
I did various memory profiling and it seems like IronPython or somewhere in DLR land unmanaged memory is not getting freed and this creeps up overtime. Managed memory seems to be getting cleared as the AppDomain is unloaded.
The C# engine itself is rather complex and interacts with MS SQL, IronPython, a Data Historian and an Asset Database. I won't go into the specifics of this as I have been able to recreate the issue by taking out all the additional components into a simple Windows Forms application.
The code I am running at the moment under a timer is:
private void RunEngine()
{
ScriptEngine pythonEngine = null;
AppDomain sandbox = null;
ScriptSource source = null;
ScriptScope scope = null;
dynamic subClass = null;
ObjectOperations ops = null;
dynamic instance = null;
dynamic result = null;
Dictionary<string, ScriptValue> scriptInputValues = GetIronPythonScriptInputAttributeValues();
Dictionary<string, ScriptValue> scriptOutputValues = GetIronPythonScriptOutputAttributes();
// Setup PythonEngine options
Dictionary<string, object> options = new Dictionary<string, object>();
//options["Debug"] = ScriptingRuntimeHelpers.True;
options["ExceptionDetail"] = ScriptingRuntimeHelpers.True;
options["ShowClrExceptions"] = ScriptingRuntimeHelpers.True;
// Create a sandbox to run the IronPython scripts in
sandbox = AppDomain.CreateDomain("IronPythonSandbox",
AppDomain.CurrentDomain.Evidence,
new AppDomainSetup() { ApplicationBase = AppDomain.CurrentDomain.BaseDirectory, ApplicationName = "IronPythonSandbox" },
new PermissionSet(PermissionState.Unrestricted));
// Create the python engine
pythonEngine = Python.CreateEngine(sandbox, options);
source = pythonEngine.CreateScriptSourceFromFile(@"\\server2\Projects\Customer\Development\Scripts\calculation.py");
var compiled = source.Compile();
scope = pythonEngine.CreateScope();
//source.Execute(scope);
compiled.Execute(scope);
subClass = scope.GetVariableHandle("Calculate");
ops = pythonEngine.Operations;
instance = ops.Invoke(subClass, scriptInputValues, scriptOutputValues);
result = instance.Unwrap();
if (scriptInputValues?.Count > 0) { scriptInputValues.Clear(); scriptInputValues = null; }
if (scriptOutputValues?.Count > 0) { scriptOutputValues.Clear(); scriptOutputValues = null; }
result = null;
instance = null;
ops = null;
subClass = null;
scope = null;
source = null;
pythonEngine?.Runtime?.Shutdown();
pythonEngine = null;
if (sandbox != null) { AppDomain.Unload(sandbox); }
sandbox = null;
}
I have stripped down the script into bare bones now to test the memory issue and it is like this and does not carry out any actual calculations as such.
import clr
import sys
# Import integration library to allow for access to the required .Net object types
sys.path.append(r"C:\Program Files\Company\RTCM Worker") # Include the path to the .Net Library
clr.AddReference('RTCM.Worker.IPy.Integration.Library.dll')
import RTCM.Worker.IPy.Integration.Library
import System
from System.Collections.Generic import Dictionary
sys.path.append(r"\\server2\Projects\Customer\Development\Scripts") # Include the path to the module
from constants import *
from sharedfunctions import *
import math
def Calculate(scriptInputValues, scriptOutputValues):
returnValue = True
try:
# Parameter validations
if returnValue: # Only proceed with the calculation if all inputs are valid
## Script logging related objects
#ENABLE_SCRIPTLOGGING = scriptOutputValues[C_EnableScriptLogging].Value
#SCRIPT_LOG = scriptOutputValues[C_ScriptLog].Value
# Get all the required input parameter values
AMB_TEMP = scriptInputValues[C_AmbientTemperature].Value
GND_AIR = scriptInputValues[C_GroundAir].Value
MAX_DESIGN_TEMP = scriptInputValues[C_MaximumDesignTemperature].Value
g = scriptInputValues[C_RatingCalculationConstants_g].Value
CONDUCTOR_DIA = scriptInputValues[C_ConductorDIA].Value
WIND_SPEED = scriptInputValues[C_WindSpeed].Value # From lookup table and no conversion needed as this is in m/s
DEFAULT_WIND_ANGLE = scriptInputValues[C_WindBearing].Value
SIGMA = scriptInputValues[C_Rating_Calculation_Constants_SIGMA].Value
CONDUCTOR_EMISSIVITY = scriptInputValues[C_ConductorEmissivity].Value
SOLAR_ABSORPTION = scriptInputValues[C_SolarAbsorption].Value
SOLAR_DIRECT = scriptInputValues[C_SolarDirect].Value
GROUND_REFLECTIVITY = scriptInputValues[C_GroundReflectivity].Value
SOLAR_DIFFUSE = scriptInputValues[C_SolarDiffuse].Value
CONDUCTOR_SKIN_EFFECT = scriptInputValues[C_ConductorSkinEffect].Value
CONDUCTOR_MAG_EFFECT = scriptInputValues[C_ConductorMAGEffect].Value
CONDUCTOR_DC_RESISTANCE = scriptInputValues[C_ConductorDCResistance].Value
CONDUCTOR_ALPHA = scriptInputValues[C_ConductorAlpha].Value
# Destroy all referenced objects
del AMB_TEMP
del GND_AIR
del MAX_DESIGN_TEMP
del g
del CONDUCTOR_DIA
del WIND_SPEED
del DEFAULT_WIND_ANGLE
del SIGMA
del CONDUCTOR_EMISSIVITY
del SOLAR_ABSORPTION
del SOLAR_DIRECT
del GROUND_REFLECTIVITY
del SOLAR_DIFFUSE
del CONDUCTOR_SKIN_EFFECT
del CONDUCTOR_MAG_EFFECT
del CONDUCTOR_DC_RESISTANCE
del CONDUCTOR_ALPHA
del scriptInputValues
del scriptOutputValues
returnValue = True
except System.Exception as ex:
returnValue = False
return returnValue;
Some screenshots of memory creeping up over time and you will notice unmanaged memory is creeping up:
I am running out of options now. Are there any other suggestions on things to try?
A few other things I tried:
- Setting LightweightScopes to true and it did not help.
- Deleting objects referenced in the IronPython script using the del keyword and it did not help.
Let me know if you want to know any additional details around my setup.