In this situation you must bypass the administration console. If you want to do that just edit the configuration file for the server, i.e. edit the server.xml for that particular server, either the application server process or the deployment manager process depending on what process suffers from the memory problem.
edit the following attributes:
jvmEntries debugArgs="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=7777" debugMode="false" disableJIT="false" genericJvmArguments="-Xquickstart" runHProf="false" verboseModeClass="false" **initialHeapSize**="512" **maximumHeapSize**="1024" verboseModeGarbageCollection="false" verboseModeJNI="false" xmi:id="JavaVirtualMachine_1183121908656"
The server.xml can be found here:
/*IBM_ROOT_DIR*/WebSphere/WAS85/AppServer/profiles/*MyProfile*/config/cells/*MyCell*/nodes/*MyNode*/servers/*MyServer*/server.xml
After the update of the server.xml just restart the server process to utilize the new memory settings.