ColdFusion scheduler threads eating CPU

Question

I've got CF10 running on a dev box, Windows 7, 64 bit. Periodically, every minute or so, the CPU usage for CF10 will spike up to 100% for about 20 seconds and come back down. It's pretty regular.

I've found it difficult to diagnose this issue. I've seen talk of client variables purges, logging, monitoring and all manner of things - but I've turned these all off to no avail.

With VisualVM, I've managed to track the issue down to the 'scheduler' threads. I have 5 of these in a waiting state. Periodically each will run, bumping up the CPU dramatically. Taking a thread dump, it seems that all these threads are calling java.io.WinNTFileSystem.getBooleanAttributes - something I've seen mentioned a few times as potentially problematic.

UPDATE: Recently I've been playing with onSessionEnd on another app, and discovered that the scheduler-x threads appear to be internal to ColdFusion - my onSessionEnd tasks always seem to run in one of these threads.

Looking in the temp folder, I can see that a lot of EH Cache folders have been made which I think are to do with query caching. The apps I have running make use of this fairly extensively. I thought clearing the temp folder out might improve performance but it has had no effect.

It's worth noting that if I start the CF service without actually calling any of my apps, the problem does not occur. That might suggest the issue is with the apps themselves, however they do not cause any issue in production - only on this box. There are no scheduled tasks set up either.

Below is an example of one of the threads causing high CPU. I'd appreciate any help in diagnosing what this thread is doing and why, as well as how to potentially stop it from using so much resources.

"scheduler-2" - Thread t@84
   java.lang.Thread.State: RUNNABLE
    at java.io.WinNTFileSystem.getBooleanAttributes(Native Method)
    at java.io.File.isDirectory(File.java:849)
    at coldfusion.watch.Watcher.accept(Watcher.java:352)
    at java.io.File.listFiles(File.java:1252)
    at coldfusion.watch.Watcher.getFiles(Watcher.java:386)
    at coldfusion.watch.Watcher.getFiles(Watcher.java:397)
    at coldfusion.watch.Watcher.getFiles(Watcher.java:397)
    at coldfusion.watch.Watcher.getFiles(Watcher.java:397)
    at coldfusion.watch.Watcher.getFiles(Watcher.java:397)
    at coldfusion.watch.Watcher.getFiles(Watcher.java:397)
    at coldfusion.watch.Watcher.getFiles(Watcher.java:397)
    at coldfusion.watch.Watcher.getFiles(Watcher.java:397)
    at coldfusion.watch.Watcher.getFiles(Watcher.java:397)
    at coldfusion.watch.Watcher.getFiles(Watcher.java:397)
    at coldfusion.watch.Watcher.checkWatchedDirectories(Watcher.java:166)
    at coldfusion.watch.Watcher.run(Watcher.java:216)
    at coldfusion.scheduling.ThreadPool.run(ThreadPool.java:211)
    at coldfusion.scheduling.WorkerThread.run(WorkerThread.java:71)

My environment:

• Win 7 64-bit
• CF10 Update 12
• JDK 1.8.0_11

The issue occurs on multiple versions of JVM - this version is currently used to make monitoring available.

My java settings:

• Min heap size: 512mb

• Max heap size: 1024mb

  -server -XX:MaxPermSize=512m -XX:+UseParallelGC -Xbatch -Dcoldfusion.home={application.home} -Dcoldfusion.rootDir={application.home} -Dcoldfusion.libPath={application.home}/lib -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random -XX:+HeapDumpOnOutOfMemoryError -Dcom.sun.management.jmxremote.port=8701 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false

I'd be lying if I said I understood what all of these settings do! Sorry if you're one of those people that believes all CF developers should be Java app stack experts. I am not. Any help, much appreciated. ;)

Answer 1

Using FusionReactor 6, I was able to solve this for us today. We were using this.javaSettings to hot load java class files. The WatchInterval from this.javaSettings uses the DirectoryWatcher at the specified watch number. In our case, I had lowered it to one second.

How I solved it: I set a breakpoint in FusionReactor and could see that it was constantly stuck scanning the directory above the one I specified in this.javasettings. This directory has enough files and subfolders, that it looks like one DirectoryWatcher was unable to finish before the next one was created. Had ColdFusion just stuck to the subfolder, I specified in this.javaSettings, it would not have been a problem.

Example:

This.javaSettings       = {
          loadPaths                 = ["\externals\lib\"]
        , loadColdFusionClassPath   = true
        , reloadOnChange            = true
        , watchInterval             = 1
};

In the above case, lib has just 5 files. However, "externals" is loaded with stuff. In the breakpoint, it was typically looking at stuff in "externals."

Answer 2

Do you have scheduled tasks running that use the CFFILE tag? They tend to be resource hogs. Spinning these into their own threads may help with the CPU spike.

another thought:

looking at the JVM,

•Min heap size: 512mb
•Max heap size: 1024mb

These establish the minimum and maximum memory available to the java virtual machine

-server -XX:MaxPermSize=512m 

This is the amount of memory dedicated to the java permanent memory generation.

you've got half of your JVM allocated memory dedicated to the permanent generation, try bumping up the maximum heap size to 2048mb. and restarting the ColdFusion service. It could go higher based on whether or not you're running a 64Bit operating system or not.