Monday, February 10, 2014

PRSTAT Linux – How to pinpoint high CPU Java VM Threads

This article will provide you with an equivalent approach, for JVM on Linux OS, of the powerful Solaris OS prstat command; allowing you to quickly pinpoint the high CPU Java VM Thread contributors.

One key concept to understand for a Java VM running on the Linux OS is that Java threads are implemented as native Threads, which results in each thread being a separate Linux process.

Ok thanks for the info but why is this related to prstat?

Well this key concept means that you don’t need a prstat command for Linux. Since each Java VM Thread is implemented as a native Thread, each Thread CPU % can simply be extracted out-of-the-box using the top command.

You still need to generate Thread Dump data of your JVM process in order to correlate with the Linux top command output.

Thanks for this explanation. Now please show me how to do this

Please simply follow the instructions below:

1)     Execute the top command (press SHIFT-H to get the Threads toggle view) or use -H option (to show all Threads) and find the PID associated with your affected  / high CPU WLS process(es) (remember, many may show up since each Java Thread is implemented as a separate Linux process)
2)     Immediately after, generate a few Thread Dump snapshots using kill –3 <PID> of the parent WLS process. Thread Dump provides you with the complete list with associated Stack Trace of each Java Thread within your JVM process
3)     Now, convert the PID(s) extracted from the top command output to HEX format
4)     The next step is to search from the Thread Dump data for a match nid=<HEX PID>
5)     The final step is to analyze the affected Thread(s) and analyze the Stack Trace so you can determine where in the code is the problem (application code, middleware itself, JDK etc.)

Example: top command captured of a Weblogic Server Java Thread running at 40% CPU utilization along with Thread Dump data via kill -3 <PID>

## top output sample 
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
...........
22111 userWLS 9 0 86616 84M 26780 S 0.0 40.1 0:00 java

  • Decimal to HEX conversion of Java Thread (native Thread) 22111 >> 0x565F
  • Now using the HEX value, we can search within the Thread Dump for the following keyword: nid=0x565F

## Thread Dump output sample Thread as per the above search criteria nid=0x565F
"ExecuteThread: '0' for queue: 'default'" daemon prio=1 tid=0x83da550 nid=0x565Fwaiting on monitor [0x56138000..0x56138870]
  at java.util.zip.ZipFile.getEntry(Native Method)
  at java.util.zip.ZipFile.getEntry(ZipFile.java:172)
  at java.util.jar.JarFile.getEntry(JarFile.java:269)
  at java.util.jar.JarFile.getJarEntry(JarFile.java:252)
  at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:989)
  at sun.misc.URLClassPath$JarLoader.findResource(URLClassPath.java:967)
  at sun.misc.URLClassPath.findResource(URLClassPath.java:262)
  at java.net.URLClassLoader$4.run(URLClassLoader.java:763)
  at java.security.AccessController.doPrivileged(AccessController.java:224)
  at java.net.URLClassLoader.findResource(URLClassLoader.java:760)
  at java.lang.ClassLoader.getResource(ClassLoader.java:444)
  at java.lang.ClassLoader.getResourceAsStream(ClassLoader.java:504)
  ............................................

In the above example, the problem was related to an excessive class loading / IO problem.


As you can see, the approach did allow us to quickly pinpoint high CPU Thread contributor but you will need to spend additional time to analyze the root cause which is now your job.

1 comment:

  1. SOFT TECH GEEKS
    Soft Tech Geeks is a chinese underground organisation of computer experts and hackers. We stay discrete in order to prevent the identity of our clients from FEDs and individuals.
    • Hacking really takes the right tools and equipment (Spywares and other necessary softwares).
    Most people really think that a hacker can just breaking into a security system just by browsing through the site.
    But if you really need a hacker we are here for you. We give you full details of our strategy on how we are going to get the job done, then you can decide if we are really or not.

    HOW WE WORK:

    • We don't ask personal questions about you and we dont give out our personal information.
    We strictly do business and don't expose you or your service to you to anyone.
    That's how discrete we are.

    OUR HACKING SERVICES:
    As long as technology is involved anything can be hacked.
    *Most people want to Hack a target's email I.D, social media websites or apps, smartphones, for many reasons such as husband cheating on wife, girlfriend cheating on boyfriend.
    *if you need to hack a particular organisation so as to sniff, delete, change a particular information or records.
    *as a parent you might need to monitor what your children are doing on social media and on their personal computers or even clone their phones to also hear their calls or see text messages on any app they chatting with, so that they don’t get into trouble.
    *if you want to hack a private domain email account(business email).
    *you want to eraze your name from court’s criminal records, perhaps you might want to Hack into the database of any government agency,
    *you want to spend someone's money by cloning another credit card to their account without bank or the user's notification,
    *some people want to mine bit coin, we can create a platform for you to mine bit coin without having to cheat anyone.
    All these are what we can get done within few hours.

    OUR ASSURANCE TO YOU:
    Fankly speaking, we always give a 100% guarantee on an job we take. Time to complete a job depends on the complication of the job and if we take your job then be rest assured that it will be done, and we give you a 100% assurance that we cover our tracks well enough, so when we do a job its like we were never there. We have also come across FAKE HACKERS claiming they will get the job done in a short time , and sharing testimony of themselves but they are all scams (AVOID THEM).
    REMEMBER THIS "AS LONG AS IT'S TECHNOLOGY IT CAN BE HACKED"

    We look forward to working for you.
    SOFT TECH GEEKS
    softtechgeeks@gmail.com

    ReplyDelete