Solaris Monitoring

Solaris Performance and Tracing Tools
Process control
System Stats
Process Tracing/
debugging
● abitrace – trace ABI interfaces
● dtrace – trace the world
● mdb – debug/control processes
● truss – trace functions and system
calls
●pgrep – grep for processes
●pkill – kill processes list
●pstop – stop processes
●prun – start processes
●prctl – view/set process
resources
●pwait – wait for process
●preap* – reap a zombie process
Process stats
● acctcom – process accounting
● busstat – Bus hardware counters
● cpustat – CPU hardware counters
● iostat – IO & NFS statistics
● kstat – display kernel statistics
● mpstat – processor statistics
● netstat – network statistics
● nfsstat – nfs server stats
● sar – kitchen sink utility
● vmstat – virtual memory stats
● cputrack / cpustat - processor hw counters
● plockstat – process locks
● pargs – process arguments
● pflags – process flags
● pcred – process credentials
● pldd – process's library dependencies
● psig – process signal disposition
● pstack – process stack dump
● pmap – process memory map
● pfiles – open files and names
● prstat – process statistics
● ptree – process tree
● ptime – process microstate times
● pwdx – process working directory
Kernel Tracing/
debugging
● dtrace – trace and monitor kernel
● lockstat – monitor locking statistics
● lockstat -k – profile kernel
● mdb – debug live and kernel cores

pargs
pflags
pcred
pldd
psig
pstack
pmap
pfiles

pwdx
pstop
prun

pwait
ptree
ptime
preap
pkill

trapstat

mpstat
vmstat
netstat
sar

cputrack cpustat

fsstat

PROC tools

pargs pflags pcred pldd psig pstack pmap pfiles pstop prun pwait ptree ptime preap

Microstates

• Fine-grained state tracking for processes/threads
• Off by default in Solaris 8 and Solaris 9
• On by default in Solaris 10
• Can be enabled per-process via /proc
• prstat -m reports microstates
• As a percentage of time for the sampling period

1. USR – user mode
2. SYS - kernel mode
3. TRP – trap handling
4. TFL – text page faults
5. DFL – data page faults
6. LCK – user lock wait
7. SLP - sleep
8. LAT – waiting for a processor (sitting on a run queue)

prstat -m

Use prstat -m LAT category, in conjunction with utilization measurements, delivered workload throughput and run queue depth (vmstat “r” column) to determine for CPU capacity planning

prstat – user summary

# prstat -t
NPROC USERNAME SIZE RSS MEMORY TIME CPU
128 root 446M 333M 1.4% 47:14:23 11%
2 measter 6600K 5016K 0.0% 0:00:07 0.2%
1 clamb 9152K 8344K 0.0% 0:02:14 0.1%
2 rmc 7192K 6440K 0.0% 0:00:00 0.1%
1 bricker 5776K 4952K 0.0% 0:00:20 0.1%
2 asd 10M 8696K 0.0% 0:00:01 0.1%
1 fredz 7760K 6944K 0.0% 0:00:05 0.1%
2 jenks 8576K 6904K 0.0% 0:00:01 0.1%
1 muffin 15M 14M 0.1% 0:01:26 0.1%
1 dte 3800K 3016K 0.0% 0:00:04 0.0%
2 adjg 8672K 7040K 0.0% 0:00:03 0.0%
3 msw 14M 10M 0.0% 0:00:00 0.0%
1 welza 4032K 3248K 0.0% 0:00:29 0.0%
2 kimc 7848K 6344K 0.0% 0:00:25 0.0%
4 jcmartin 13M 9904K 0.0% 0:00:03 0.0%
1 rascal 17M 16M 0.1% 0:02:11 0.0%
1 rab 3288K 2632K 0.0% 0:02:11 0.0%
1 gjmurphy 3232K 2392K 0.0% 0:00:00 0.0%
1 ktheisen 15M 14M 0.1% 0:01:16 0.0%
1 nagendra 3232K 2400K 0.0% 0:00:00 0.0%
2 ayong 8320K 6832K 0.0% 0:00:02 0.0%
Total: 711 processes, 902 lwps, load averages: 3.84, 4.30, 4.37

Solaris iostat

● Wait: number of threads queued for I/O ● Actv: number of threads performing I/O ● wsvc_t: Average time spend waiting on queue ● asvc_t: Average time performing I/O ● %w: Only useful if one thread is running on the entire machine – time spent waiting for I/O ● %b: Device utilization – only useful if device can do just 1 I/ O at a time (invalid for arrays etc...)

# iostat -xnz
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
687.8 0.0 38015.3 0.0 0.0 1.9 0.0 2.7 0 100 c0d0
Queue Performing I/O
wait svc

New Formatting flags -C, -l, -m, -r, -s, -z, -T
● -C: report disk statistics by controller
● -l n: Limit the number of disks to n
● -m: Display mount points (most useful with -p)
● -r: Display data n comma separated format
● -s: Suppress state change messages
● -z: Suppress entries with all zero values
● -T d|u Display a timestamp in date (d) or unix time_t (u)