WebLogic stuck on internal logging

Question

After executing performance testing, several nodes in Web Logic cluster were stuck with same reason:

"[ACTIVE] ExecuteThread: '201' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock java.util.WeakHashMap@8632059 BLOCKED

            weblogic.logging.WLLogger.normalizeLogRecord(WLLogger.java:60)

            weblogic.logging.WLLogger.log(WLLogger.java:37)

            weblogic.diagnostics.debug.DebugLogger.log(DebugLogger.java:231)

            weblogic.diagnostics.debug.DebugLogger.debug(DebugLogger.java:204)

            weblogic.jdbc.wrapper.JDBCWrapperImpl.trace(JDBCWrapperImpl.java:169)

            weblogic.jdbc.wrapper.ResultSet.preInvocationHandler(ResultSet.java:83)

            weblogic.jdbc.wrapper.ResultSet_oracle_jdbc_driver_OracleResultSetImpl.getLong(Unknown Source)

There is total 126 threads blocked on the same

waiting for lock java.util.WeakHashMap@8632059

I really wonder, what could cause this strange behaviour, I was expecting troubles in database performance or lack of jdbc connections, but not logging issues. If so, is it possible to disable this logging somehow?

Other node also had

"[STUCK] ExecuteThread: '206' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.logging.FileStreamHandler@153405fb BLOCKED

            com.bea.logging.RotatingFileStreamHandler.publish(RotatingFileStreamHandler.java:75)

            java.util.logging.Logger.log(Logger.java:610)

            com.bea.logging.BaseLogger.log(BaseLogger.java:51)

            weblogic.logging.WLLogger.log(WLLogger.java:42)

with 15 blocked threads. The underlying method in all cases does small jpa query, selecting user by login from db due to invalidated cache.

What version of WebLogic is this? If you leave it for a while do these threads become unblocked? — Steve C
WebLogic 12.1.2. I can't leave threads, since jdbc connections blocked inside stuck methods prevent serving other requests in performance test (ofc it leads to test fail) — rdo
I understand where you're coming from, my question was just for diagnosing what's going on. I think your problem is that you have JDBC tracing turned on and it's bottlenecking on a couple of normally innocuous synchronised blocks. If you turn the tracing off it should never hit those blocks in the first place. — Steve C

jlego jlego · Accepted Answer · 2018-05-16T12:18:39

I've hit exactly the same problem. Few thread dumps indicated that Weblogic is stuck in some kind of internal debug logging:

 - weblogic.diagnostics.debug.DebugLogger.log(java.util.logging.Level, java.lang.String, java.lang.Throwable) @bci=56, line=246 (Compiled frame)
 - weblogic.diagnostics.debug.DebugLogger.debug(java.lang.String) @bci=6, line=211 (Compiled frame)

Internal debug levels can be configured in Admin Console in Environment > Servers > MyServer > Debug. In my case I discovered that all possible scopes were enabled. After disabling debug logging and restarting server problems disappeared.

WebLogic stuck on internal logging

1 Answers