Few commands to troubleshoot:

Networking slowness

ping -c 3 -M do -s 30000 -T IP


Grep based on time range
grep -v -e "^[2014-03-06T08:[30-59]" soa_server2-diagnostic-1297.log |more

Find number of threads running in the system

cat /proc/PID/status

ps -eLo pid,cmd,nlwp

Heap dump

/u01/app/jdk1.6.0_31/bin/jmap -dump:format=b,file=heapdump.hprof 8440

Top running process consuming high CPU

top –H –b –p PID

Strace monitors the system calls and signals of a specific program

/usr/bin/strace –s 200 –o /u01/tmp/some.txt –fp <PID>

Search receive from and socket descriptor from the output file

List of open files: Lsof

/usr/bin/lsof  |grep port

Sar reports extraction:

LC_ALL=C sar -A -f /var/log/sa/sa09 >> sar_node1_nov9.txt

You can use ksar tool and generate a nice PDF from the output file.

Zombie process: 

 ps -lA |grep '^. Z'

4 Z     0  4332  3938  0  85   0 -     0 exit   ?        00:00:00 pwck <defunct>

4 Z     0  4339  3938  0  80   0 -     0 exit   ?        00:00:00 test <defunct>

4 Z     0  4739  3938  0  82   0 -     0 exit   ?        00:00:00 pwck <defunct>

0 Z     0 27064  3938  0  75   0 -     0 exit   ?        00:00:00 ldapsearch <defunct>

$ ps axo stat,ppid,comm |grep -w defunct

Zs    3938 pwck <defunct>

Zs    3938 test <defunct>

Zs    3938 pwck <defunct>

Zs    3938 ldapsearch <defunct>

$ ps -ef |grep 3938

Pmap command reports the memory map of a process:
cat /proc/pid/status or whatever

pmap -x <process pid>

Java heap dump

I recently worked on a project where I was faced with "out of memory" problem. Exception appeared usually after the application has worked for several weeks. I solved the problem with VisualVM tool that ships with Java 1.6 JDK.
VisualVM tool provides a direct connection to any Virtual Machine locally or remote, or can load heap dump or thread dump from a file.

To dump heap memory to a file type in console(linux):

1.             Get java process id
ps -eaf | grep java

2.             Dump heap memory to a file using jmap ("jmap prints shared object memory maps or heap memory details of a given process or core file or a remote debug server")
jmap -heap:format=b PID (where PID i id of your java process)

3.             Run jvisualvm
jvisualvm -J-Xmx512M

4.             Load heap.bin file

5.             Find biggest objects by retained size. After file loads you can see every object that was in memory at the time the dump was taken.Click on find biggest objects by retained size and you will get hierarchy of biggest objects in memory.

ps – Displays The Processes

ps command will report a snapshot of the current processes. To select all processes use the -A or -e option:
# ps -A
Sample Outputs:

ps -A
PID TTY TIME CMD
1 ttyp0 00:00:22 init
89 ? 00:00:00 init-logger
142 ? 00:00:00 udevd
634 ? 00:00:00 syslog-ng
635 ? 00:20:39 syslog-ng
718 ? 00:00:00 zabbix_agentd
720 ? 00:04:02 zabbix_agentd
736 ? 00:00:00 xinetd
754 ? 00:00:00 mysqld_safe
784 ? 21:08:05 mysqld
788 pts/2 00:00:00 zsh
820 ? 00:00:01 logger
826 ? 00:00:00 httpd
827 ? 00:00:34 httpd
1154 ? 00:00:00 gam_server
1633 ? 00:05:30 httpd
1642 ? 00:00:05 pure-ftpd
....
32656 ? 00:00:00 exim
32659 ? 00:00:02 php5
32660 ? 00:00:00 php5
32691 ? 00:00:00 gam_server
32698 ? 00:00:01 php5
32700 ? 00:00:00 php5
32704 ? 00:00:00 exim
32705 ? 00:00:00 cpsrvd-ssl
32707 pts/0 00:00:00 ps

Now for some switches that will allow you to do a lot more with this command:

Show Long Format Output

# ps -Al

To turn on extra full mode (it will show command line arguments passed to process):

# ps -AlF

To See Threads ( LWP and NLWP)

# ps -AlFH

To See Threads After Processes

# ps -AlLm

Print All Process On The Server

# ps ax
# ps axu

Print A Process Tree

# ps -ejH
# ps axjf
# pstree

Print Security Information

# ps -eo euser,ruser,suser,fuser,f,comm,label
# ps axZ
# ps -eM

See Every Process Running As User Vivek

# ps -U vivek -u vivek u

Set Output In a User-Defined Format

# ps -eo pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:14,comm
# ps axo stat,euid,ruid,tty,tpgid,sess,pgrp,ppid,pid,pcpu,comm
# ps -eopid,tt,user,fname,tmout,f,wchan

Display Only The Process IDs of httpd (apache)

# ps -C httpd -o pid=
OR
# pgrep httpd
OR
# pgrep -u vivek php-cgi

Display The Name of PID 58721

# ps -p 58721 -o comm=

Find Out The Top 10 Memory Consuming Process

# ps -auxf | sort -nr -k 4 | head -10

Find Out top 10 CPU Consuming Process

# ps -auxf | sort -nr -k 3 | head -10