We have our Application (Spring Boot App) deployed and up and running in PCF for couple of days, the requests are processed and container memory is increased gradually and reaching 80 or 90 %, but is not reduced at all. At one point the app is crashed.
CPU behaviour seems fine, when there is load it is spiking and dropping again. It works fine in local Machine.My Guess is that the GC is not happening properly within PCF Container.
Have anyone faced this issue, Could anyone help us.
It works fine in local Machine.
This is almost certainly an apples-to-oranges comparison (i.e. not a valid comparison). It's quite likely that there are many things that differ between your local environment and the app running on Cloud Foundry. All of those differences can affect the way your app runs.
My suggestion would be to use the cf local cf cli plugin to run your app locally. This will let you stage & run the app locally in docker containers. It has the benefit of staging the app with the same buildpack that will run to stage in on Cloud Foundry and it will also run your app in a container with the same memory limits as your app running on Cloud Foundry. It's still not exactly the same, but it gets you a lot closer.
https://github.com/cloudfoundry-incubator/cflocal
Aside from that:
Always make sure you're on the latest Java buildpack version. No one should still be using 3.x at the time this is written. If you're running 3.x & seeing crashes, upgrade and there's a very good chance they'll just go away.
Version 4.x has a lot of improvements to the way memory usage is calculated so it's much better at preventing the JVM from exceeding the memory limit in your container and crashing the app. There have even been incremental improvements in the 4.x versions, so that's why you want to run the most recent one available.
If you're seeing OutOfMemoryError exceptions, this just means you're running out of heap or some other segment of memory. You likely just need to increase the memory limit for your app and restage. The Java buildpack will recalculate the different memory regions based on your new memory limit and you'll end up with more memory to use. Alternatively, you could set JAVA_OPTS and adjust the JVM's memory settings manually, but that's not recommended unless you are an expert.
If increasing the memory doesn't help or you've increased multiple times and still run out of memory, you probably have a memory leak. Enlist the help of an APM tool or a Profiler to debug your leak.
If the container is crashing with exit 137 (means your app exceeded the memory limit set) & you're running the latest Java buildpack, look to see if your app is using native memory (i.e. JNI) for anything, even from dependent libraries. This can grow unbounded and cause you to go above your memory limit.
Similar to #3, look at the number of threads being used by your app. The Java buildpack estimates for a reasonable number of threads, but it cannot limit the number of threads the JVM or your app can create. If the app goes above the number of threads the Java buildpack estimates then you can go above the memory limit and cause your app to crash. To resolve this, you can increase the estimated number of threads, you can increase the memory limit or decrease/cap thread usage in your app.
If all else fails, you can enable Java NMT (native memory tracing). This is a little tricky, but it can give you a pretty good picture of where the JVM is using memory. Grab some snapshots during the lifetime of your app and you can usually see which segment is growing and which is causing the app to crash. There's a couple ways to do this to an app running on CF, see here and here for some details.
