java.lang.OutOfMemoryError: unable to create new native thread error using ChromeDriver and Chrome through Selenium in Spring boot

2
votes

We have selenium based web application developed using spring boot. The server is located as VM Instance at google cloud server.

We have a thread base mechanism using Executor. Using selenium we open a chrome browser (headless) to perform operation and for each operation we create new thread.

After facing outOfMemory error, if we restart cloud instance then within 1 hour it breaks again with the same error.

Please find below the snippet which we used to create a new instance of executor service for each request.

executorService = Executors.newFixedThreadPool(1);
Future<Object> futureDetails = executorService.submit(new Callable<Object>() {

        @Override
        public Object call() throws Exception {

            Object response = client.getDetails(number);

            return response;
        }
    });

    executorService.shutdown();

We have (16 vCPUs, 64 GB memory) server configuration.

Please find below the details of the version.

Spring Boot - 2.0.0.RELEASE

Selenium - 3.9.1

Linux - Debian 4.9.130-2 (2018-10-27) x86_64 GNU/Linux

Chrome Driver - 2.35.528139 (47ead77cb35ad2a9a83248b292151462a66cd881)

Google Chrome - 70.0.3538.110

JDK - 1.8.0_232

Please find below resource report using: ulimit -a.

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 257648
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 10240
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 257648
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Also, I did check the memory using: free -m

              total        used        free      shared  buff/cache   available
Mem:          64433        9479       54490         279         463       54132
Swap:             0           0           0

Please find below the error stack trace:

org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: unable to create new native thread at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1006) ~[spring-webmvc-5.0.4.RELEASE.jar!/:5.0.4.RELEASE] at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925) ~[spring-webmvc-5.0.4.RELEASE.jar!/:5.0.4.RELEASE] at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:978) ~[spring-webmvc-5.0.4.RELEASE.jar!/:5.0.4.RELEASE] at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:881) ~[spring-webmvc-5.0.4.RELEASE.jar!/:5.0.4.RELEASE] at javax.servlet.http.HttpServlet.service(HttpServlet.java:661) [tomcat-embed-core-8.5.28.jar!/:8.5.28] at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:855) ~[spring-webmvc-5.0.4.RELEASE.jar!/:5.0.4.RELEASE] at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) [tomcat-embed-core-8.5.28.jar!/:8.5.28] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [tomcat-embed-core-8.5.28.jar!/:8.5.28] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.28.jar!/:8.5.28] at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-embed-websocket-8.5.28.jar!/:8.5.28] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.28.jar!/:8.5.28] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.28.jar!/:8.5.28] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:320) [spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE] at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127) [spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE] at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91) [spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) [spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE] at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119) [spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) [spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE] at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137) [spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) [spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE] at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) [spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) [spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE] at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170) [spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) [spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE] at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) [spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) [spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE] at org.springframework.security.oauth2.provider.authentication.OAuth2AuthenticationProcessingFilter.doFilter(OAuth2AuthenticationProcessingFilter.java:176) [spring-security-oauth2-2.2.1.RELEASE.jar!/:na] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) [spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE] at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116) [spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) [spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE] at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66) [spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) [spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE] at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105) [spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) [spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE] at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56) [spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) [spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE] at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:215) [spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE] at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178) [spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE] at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:357) [spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE] at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:270) [spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.28.jar!/:8.5.28] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.28.jar!/:8.5.28] at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) [spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.28.jar!/:8.5.28] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.28.jar!/:8.5.28] at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109) [spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.28.jar!/:8.5.28] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.28.jar!/:8.5.28] at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81) [spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.28.jar!/:8.5.28] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.28.jar!/:8.5.28] at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) [spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.28.jar!/:8.5.28] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.28.jar!/:8.5.28] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.28.jar!/:8.5.28] at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199) [tomcat-embed-core-8.5.28.jar!/:8.5.28] at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-8.5.28.jar!/:8.5.28] at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496) [tomcat-embed-core-8.5.28.jar!/:8.5.28] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [tomcat-embed-core-8.5.28.jar!/:8.5.28] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) [tomcat-embed-core-8.5.28.jar!/:8.5.28] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [tomcat-embed-core-8.5.28.jar!/:8.5.28] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [tomcat-embed-core-8.5.28.jar!/:8.5.28] at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803) [tomcat-embed-core-8.5.28.jar!/:8.5.28] at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-8.5.28.jar!/:8.5.28] at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790) [tomcat-embed-core-8.5.28.jar!/:8.5.28] at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459) [tomcat-embed-core-8.5.28.jar!/:8.5.28] at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-8.5.28.jar!/:8.5.28] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_241] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_241] at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.5.28.jar!/:8.5.28] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_241] Caused by: java.lang.OutOfMemoryError: unable to create new native thread

Please help me to understand the issue and root cause of this problem. It would be better if you can suggest a solution as well.

Thanks in advance.

3
javamultithreadingspring-bootseleniumout-of-memory
Selenium tests can definitely become memory hogs, especially if a test run goes for a long time. I wonder if you can pass flags to the test runner to allocate more memory to that Java process. Although that might just delay the OutOfMemory error. – Greg Burghardt
stackoverflow.com/questions/16789288/… – hrdkisback

3 Answers

2
votes

This error message...

java.lang.OutOfMemoryError: unable to create new native thread

...implies that JVM was unable to create any new native thread as your system have run OutOfMemory

OutOfMemoryError

Out of Memory error messages can appear when you attempt to start new programs or you try to use programs that are already running, even though you still have plenty of physical and pagefile memory available. When you run a java program or java application, allocates some memory to the JVM. JVM divides this memory into two parts. One is the Stack Memory and another is the Heap Memory. Stack Memory is used for execution of methods while Heap Memory is used to store the objects. When the heap becomes full, JVM throws java.lang.OutOfMemoryError.

java.lang.OutOfMemoryError

java.lang.OutOfMemoryError is thrown when the Heap Memory is full and JVM is unable to allocate the memory to new objects. As the objects you create in Java are stored in the Heap Memory, so when ever the objects are no more required, they must be removed from the memory. Garbage Collector automatically removes the unwanted objects from the Heap Memory. If your objects have live references, garbage collector doesn’t remove them. It removes only those objects which don’t have live references. in this process if at any point of time there is no space left for new objects in the heap memory then JVM will throw java.lang.OutOfMemoryError.

This usecase

As per your code trials, you have implemented Multithreading using ThreadPoolExecutor to create and use a thread for each request while opening a browsing context.

Multithreading using Selenium

As you mentioned, within 1 hour it breaks agian with the same error, I suspect this due to the fact that WebDriver is not thread-safe. Having said that, if you can serialize access to the underlying driver instance, you can share a reference in more than one thread. This is not advisable. But you can always instantiate one WebDriver instance for each thread.

Ideally the issue of thread-safety isn't in your code but in the actual browser bindings. They all assume there will only be one command at a time (e.g. like a real user). But on the other hand you can always instantiate one WebDriver instance for each thread which will launch multiple browsing tabs/windows. Till this point it seems your program is perfect.

Now, different threads can be run on same Webdriver, but then the results of the tests would not be what you expect. The reason behind is, when you use multi-threading to run different tests on different tabs/windows a little bit of thread safety coding is required or else the actions you will perform like click() or send_keys() will go to the opened tab/window that is currently having the focus regardless of the thread you expect to be running. Which essentially means all the test will run simultaneously on the same tab/window that has focus but not on the intended tab/window.

Additional Consideration

However another issue is the incompatibility between the version of the binaries you are using as follows:

  • You are using chromedriver=2.35
  • Release Notes of chromedriver=2.35 clearly mentions the following :

Supports Chrome v62-64

  • You are using chrome=70.0
  • Release Notes of ChromeDriver vv2.44 clearly mentions the following :

Supports Chrome v69-71

  • Your Selenium Client version is 3.9.1 which is almost 2 years older.

So there is a clear mismatch between Selenium Client v3.9.1 , ChromeDriver v2.35 and the Chrome Browser v70.0

Solution

Ensure that:

  • JDK is upgraded to current levels JDK 8u241.
  • Selenium is upgraded to current levels Version 3.141.59.
  • ChromeDriver is updated to current ChromeDriver v80.0 level.
  • Chrome is updated to current Chrome Version 80.0 level. (as per ChromeDriver v80.0 release notes)
  • If your base Web Client version is too old, then uninstall it through Revo Uninstaller and install a recent GA and released version of Web Client.
  • Take a System Reboot.
  • Always invoke driver.quit() within tearDown(){} method to close & destroy the WebDriver and Web Client instances gracefully.

tl; dr

How to set memory limit for OOM Killer for chrome?

1
votes

The amount of RAM your server has doesn't matter if you don't allocate that memory to your java process.

If you run your application with (any variant of) java -jar command, then add a maximum allowed memory parameter -xmx (check this question). If the application runs inside Tomcat (or any application server), you'll need to increase the maximum allowed memory of it (check its parameters and docs).

0
votes

I ran into this problem, too. The problem was that if a submitted or an executed pool thread accessed outside the pool something that requires "too much" memory. If you have thread that does such accesses you should run it as a REAL thread, not as a Runnable in an ExecutorService pool.