Accessing Kafka stream's KTable underlying RocksDB memory usage

1
votes

I have a kafka stream app that currently takes 3 topics and aggregates them into a KTable. This app resides inside a scala microservice on marathon which has been allocated 512 MB memory to work with. After implementing this, I've noticed that the docker container running the microservice eventually runs out of memory and was trying to debug the cause.

My current theory (whilst reading the sizing guide https://docs.confluent.io/current/streams/sizing.html) is that over time, the increasing records stored in the KTable and by extension, the underlying RocksDB, is causing the OOM for the microservice. Is there any way to find out the memory used by the underlying default RocksDB implementation?

1
apache-kafkaapache-kafka-streamsrocksdb
You can use a memory profiler such as Mission Control to find out which data structures are holding memory in the JVM. - ssice
Also, are you running the app on kubernetes or other container orchestrator with constrained memory limits without having limited the heap size of your JVM? Because the heap size will outgrow your limits if you don't set those accordingly. - ssice
Hi @ssice, the app is running inside a docker container in marathon. I currently don't have remote access to my device and can't immediately answer the question off the top of my head about the heap size setting for the JVM. I will get back to you on that ASAP. - pwon460
Maybe... But you did ask "Is there any way to find out the memory used by the underlying default RocksDB implementation?" - OneCricketeer
Hi @ssice, the heap size has been limited to 40% of the allocated 512MB which is roughly 204MB. That being said, I haven't seen any OOM errors thrown by the JVM as the scala code would have propagated the fatal error all the way up, resulting in failure of service health checks. What I'm seeing is docker killing the service as the container exceeded the allocated 512MB instead of marathon killing the service due to a health check failure. - pwon460

1 Answers

2
votes

In case anyone runs into a similar issue, setting the environment variable MALLOC_ARENA_MAX=2 seems to have fixed it for me. For a more detailed explanation as to why, please refer to section "Why memory allocators make a difference?" and "Tuning glibc" here: https://github.com/prestodb/presto/issues/8993.