Solaris Monitoring

Solaris Performance and Tracing Tools
Process control
System Stats
Process Tracing/
debugging
● abitrace – trace ABI interfaces
● dtrace – trace the world
● mdb – debug/control processes
● truss – trace functions and system
calls
●pgrep – grep for processes
●pkill – kill processes list
●pstop – stop processes
●prun – start processes
●prctl – view/set process
resources
●pwait – wait for process
●preap* – reap a zombie process
Process stats
● acctcom – process accounting
● busstat – Bus hardware counters
● cpustat – CPU hardware counters
● iostat – IO & NFS statistics
● kstat – display kernel statistics
● mpstat – processor statistics
● netstat – network statistics
● nfsstat – nfs server stats
● sar – kitchen sink utility
● vmstat – virtual memory stats
● cputrack / cpustat - processor hw counters
● plockstat – process locks
● pargs – process arguments
● pflags – process flags
● pcred – process credentials
● pldd – process's library dependencies
● psig – process signal disposition
● pstack – process stack dump
● pmap – process memory map
● pfiles – open files and names
● prstat – process statistics
● ptree – process tree
● ptime – process microstate times
● pwdx – process working directory
Kernel Tracing/
debugging
● dtrace – trace and monitor kernel
● lockstat – monitor locking statistics
● lockstat -k – profile kernel
● mdb – debug live and kernel cores

pargs
pflags
pcred
pldd
psig
pstack
pmap
pfiles

pwdx
pstop
prun

pwait
ptree
ptime
preap
pkill

trapstat

mpstat
vmstat
netstat
sar

cputrack cpustat

fsstat

Microstates

• Fine-grained state tracking for processes/threads
• Off by default in Solaris 8 and Solaris 9
• On by default in Solaris 10
• Can be enabled per-process via /proc
• prstat -m reports microstates
• As a percentage of time for the sampling period

1. USR – user mode
2. SYS - kernel mode
3. TRP – trap handling
4. TFL – text page faults
5. DFL – data page faults
6. LCK – user lock wait
7. SLP - sleep
8. LAT – waiting for a processor (sitting on a run queue)

prstat -m

Use prstat -m LAT category, in conjunction with utilization measurements, delivered workload throughput and run queue depth (vmstat “r” column) to determine for CPU capacity planning

prstat – user summary

# prstat -t
NPROC USERNAME SIZE RSS MEMORY TIME CPU
128 root 446M 333M 1.4% 47:14:23 11%
2 measter 6600K 5016K 0.0% 0:00:07 0.2%
1 clamb 9152K 8344K 0.0% 0:02:14 0.1%
2 rmc 7192K 6440K 0.0% 0:00:00 0.1%
1 bricker 5776K 4952K 0.0% 0:00:20 0.1%
2 asd 10M 8696K 0.0% 0:00:01 0.1%
1 fredz 7760K 6944K 0.0% 0:00:05 0.1%
2 jenks 8576K 6904K 0.0% 0:00:01 0.1%
1 muffin 15M 14M 0.1% 0:01:26 0.1%
1 dte 3800K 3016K 0.0% 0:00:04 0.0%
2 adjg 8672K 7040K 0.0% 0:00:03 0.0%
3 msw 14M 10M 0.0% 0:00:00 0.0%
1 welza 4032K 3248K 0.0% 0:00:29 0.0%
2 kimc 7848K 6344K 0.0% 0:00:25 0.0%
4 jcmartin 13M 9904K 0.0% 0:00:03 0.0%
1 rascal 17M 16M 0.1% 0:02:11 0.0%
1 rab 3288K 2632K 0.0% 0:02:11 0.0%
1 gjmurphy 3232K 2392K 0.0% 0:00:00 0.0%
1 ktheisen 15M 14M 0.1% 0:01:16 0.0%
1 nagendra 3232K 2400K 0.0% 0:00:00 0.0%
2 ayong 8320K 6832K 0.0% 0:00:02 0.0%
Total: 711 processes, 902 lwps, load averages: 3.84, 4.30, 4.37

Solaris iostat

● Wait: number of threads queued for I/O ● Actv: number of threads performing I/O ● wsvc_t: Average time spend waiting on queue ● asvc_t: Average time performing I/O ● %w: Only useful if one thread is running on the entire machine – time spent waiting for I/O ● %b: Device utilization – only useful if device can do just 1 I/ O at a time (invalid for arrays etc...)

# iostat -xnz
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
687.8 0.0 38015.3 0.0 0.0 1.9 0.0 2.7 0 100 c0d0
Queue Performing I/O
wait svc

New Formatting flags -C, -l, -m, -r, -s, -z, -T
● -C: report disk statistics by controller
● -l n: Limit the number of disks to n
● -m: Display mount points (most useful with -p)
● -r: Display data n comma separated format
● -s: Suppress state change messages
● -z: Suppress entries with all zero values
● -T d|u Display a timestamp in date (d) or unix time_t (u)