The dumpthreads script uses the Java jstack utility to obtain a thread dump from the Rhino node process.
Thread dumps are written to the file $RHINO_HOME/work/threaddump.log.<timestamp> where <timestamp> is the current date and time.
Obtaining a thread dump using jstack is near instantaneous and is safe to execute against production systems under load.
Thread dumps can be useful for diagnosing issues such as stuck threads or deadlocks in the Rhino node.
dumpthreads is also used internally by Rhino to produce diagnostic thread dumps in certain scenarios such as:
-
when a watchdog condition fails, such as a savanna group heartbeat failure or a stuck event processing thread
-
when the activity handler cleanup thread detects the oldest generation still required in the database is older than the configured warning threshold.
Below is an example of partial thread-dump output:
"Timer-0" #14 daemon prio=5 os_prio=0 cpu=0.19ms elapsed=41068.14s tid=0x00007fe740a73000 nid=0x5862a in Object.wait() [0x00007fe6fb3fd000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(java.base@11.0.25/Native Method)
- waiting on <0x00000000e1400178> (a java.util.TaskQueue)
at java.lang.Object.wait(java.base@11.0.25/Object.java:328)
at java.util.TimerThread.mainLoop(java.base@11.0.25/Timer.java:527)
- waiting to re-lock in wait() <0x00000000e1400178> (a java.util.TaskQueue)
at java.util.TimerThread.run(java.base@11.0.25/Timer.java:506)
"StageWorker/RTM/1" #82 prio=5 os_prio=0 cpu=1.07ms elapsed=41043.33s tid=0x00007fe6c4ca0800 nid=0x58690 waiting on condition [0x00007fe6ec7fe000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.25/Native Method)
- parking to wait for <0x00000000e189a270> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.25/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.25/AbstractQueuedSynchronizer.java:2081)
at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.25/LinkedBlockingQueue.java:433)
at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.25/ThreadPoolExecutor.java:1054)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.25/ThreadPoolExecutor.java:1114)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.25/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.25/Thread.java:829)
at com.opencloud.staging.FIFOQueue$1$1.run(0.0.1-SNAPSHOT:46)
