Logstash in Docker not responsive

Question

I need to migrate an ElasticSearch/Logstash process from Windows to Docker. This process works fine in Windows (Elasticsearch and Logstash are services where Logstash reads an Oracle database to feed ElasticSearch). The problem is that when I start the Logstash container, Docker becomes unresponsive and extremely slow, for example docker ps takes one minute. It takes another minute to kill the logstash container. I'm running this on Windows 10 Pro with Docker Desktop and followed these steps.

I downloaded the two images (elasticsearch:7.5.1 and logstash:7.5.1) and started the containers with

docker run -d -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" elasticsearch:7.5.1

and

docker run --name cont_logstash -d -e LIB=/usr/share/logstash/pipeline/ojdbc8.jar -e 
DRIVER=Java::oracle.jdbc.driver.OracleDriver -e CONN=jdbc:oracle:thin:@//172.0.0.2:51521/XE 
-e USER=userdb -e PASSWORD=xxx -v vol_logstash:/usr/share/logstash/pipeline/ logstash:7.5.1

Note that I created a volume to store the logstash.conf file and the ojdbc8.jar Oracle JDBC driver. If I create a bash in the Logstash container, I can see these files in /usr/share/logstash/pipeline/.

Not only is Docker very slow, but Logstash doesn't work at all. If I run Logstash interactively with -it I get this (note that there are no warnings about the setup):

C:\docker>docker run --name cont_logstash -it -e LIB=/usr/share/logstash/pipeline/ojdbc8.jar -e DRIVER=Java::oracle.jdbc.driver.OracleDriver -e CONN=jdbc:oracle:thin:@//localhost:51521/XE -e USER=userdb -e PASSWORD=xxx -v vol_logstash:/usr/share/logstash/pipeline/ logstash:7.5.1

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.8.0.jar) to field java.io.FileDescriptor.fd WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release

According to the documentation Logstash on Docker logs in the console, but I don't see any errors on the screen. Any ideas how to approach this?

UPDATE

I run other containers with no issues, including a 8.4 Gb Oracle database.

This is what I see with the command docker stats:

After I cancel the logstash container, the stats are populated.

How your docker stats looks like when "Docker is very slow"? — Jan Garaj
Are you sure that used Linux VM (Hyper-V in the Docker Desktop case) has enough memory? — Jan Garaj
@Jan see the update, that's what I see when I do stats. I can run a 8.4 Gb Oracle database in the same docker, so it's not lack of resources. My laptop has 32Gb memory and 2Tb hard disk. — ps0604
Laptop can have 32GB, but used Hyper-V machine may have configured lower limits (I guess default is 2GB). All Java containers are memory hungry, because you didn't cap Java memories (all -XX:*RAM* configs). So how much memory is consumed by containers without logstash (you can see that in docker stats) and how much memory is configured for VM (Docker’s system tray->Settings->Advanced)? — Jan Garaj
Yes, changing memory limit to 12Gb and swap to 2Gb fixed the problem — ps0604

Jan Garaj Jan Garaj · Accepted Answer · 2020-01-22T18:00:02

Laptop can have 32GB, but used Hyper-V machine may have configured lower limits (I guess default is 2GB). All Java containers are memory hungry, because you didn't cap Java memories (all -XX:*RAM* configs). So how much memory is consumed by containers without logstash (you can see that in docker stats) and how much memory is configured for VM (Docker’s system tray->Settings->Advanced)?

Logstash in Docker not responsive

2 Answers