sar, sadc & sa1

The venerable system activity reporter or sar can show us a wealth of performance statistics related to CPU, I/O, memory, and a lot more.

As is the case with other commands, if we run it sar without arguments by default it should show CPU statistics since last boot (-u flag is the default one as we shall see later). However, if the service sysstat is running…

root:/tmp> systemctl status sysstat
● sysstat.service ­- Resets System Activity Logs
.  Loaded: loaded (/usr/lib/systemd/system/sysstat.service; enabled)
.  Active: active (exited) since Thu 2015­10­15 15:39:30 BST; 1 weeks 0 days ago
.Main PID: 1102 (code=exited, status=0/SUCCESS)
.  CGroup: /system.slice/sysstat.service
.
Oct 15 15:39:30 envy17 systemd[1]: Started Resets System Activity Logs.

… we would see something like this …

root:/etc/sysconfig> sar
.
09:25:03 AM      CPU  %user  %nice  %system  %iowait  %steal  %idle
09:30:07 AM      all   4.20   0.28     1.67     2.38    0.00  91.46
09:40:05 AM      all   3.04   0.00     0.57     2.19    0.00  94.20
09:50:14 AM      all   3.23   0.00     0.59     2.60    0.00  93.58
10:00:15 AM      all   3.11   0.00     0.51     2.35    0.00  94.03
10:10:15 AM      all   3.20   0.01     0.55     2.45    0.00  93.79
10:20:15 AM      all   3.22   0.00     0.51     2.65    0.00  93.62
10:30:15 AM      all   3.15   0.00     0.50     2.40    0.00  93.94
10:40:15 AM      all   3.10   0.00     0.48     2.72    0.00  93.70
10:50:15 AM      all   3.10   0.00     0.47     2.57    0.00  93.86
11:00:15 AM      all   2.95   0.00     0.44     2.32    0.00  94.29
11:10:25 AM      all   3.15   0.02     0.50     2.85    0.00  93.47
11:20:05 AM      all   3.08   0.00     0.49     2.74    0.00  93.69
11:30:05 AM      all   3.27   0.05     0.62     2.76    0.00  93.30
Average:         all   3.16   0.02     0.53     2.52    0.00  93.77

Why is that? When the sysstat service is started at boot time, it runs the /usr/lib64/sa/sa1 command with the options laid out in the configuration file /etc/sysconfig/sysstat:

root:/etc/sysconfig> systemctl show sysstat | grep -­i execstart
ExecStart={ path=/usr/lib64/sa/sa1 ; argv[]=/usr/lib64/sa/sa1 ­­–boot ; ignore_errors=no ;
start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
root:~> cat /etc/sysconfig/sysstat
# sysstat­11.0.0 configuration file.
.
# How long to keep log files (in days).
# If value is greater than 28, then log files are kept in
# multiple directories, one for each month.
HISTORY=90
.
# Compress (using gzip or bzip2) sa and sar files older than (in days):
COMPRESSAFTER=7
.
# Parameters for the system activity data collector (see sadc manual page)
# which are used for the generation of log files.
SADC_OPTIONS=”­D ­S ALL”
.
# Directory where sa and sar files are saved.
SA_DIR=/var/log/sa
.
# Compression program to use.
ZIP=”bzip2″

The sa1 process dumps the binary data onto files in the /var/log/sa  directory and rotates them every day:

root:/etc/sysconfig> ls ­-l /var/log/sa
total 7144
-rw­-r–­­r–­­. 1 root root 218364 Oct  1 16:00 sa01
-rw­-r­­–r­­–. 1 root root 207724 Oct  2 15:50 sa02
-rw-­r­­–r­­–. 1 root root 197060 Oct  5 15:50 sa05
-rw­-r­­–r­–­. 1 root root 117440 Oct  6 15:20 sa06
-rw­-r­­–r­­–. 1 root root 181136 Oct  7 16:30 sa07
-rw­-r­­–r­­–. 1 root root 223600 Oct  8 15:50 sa08
-rw­-r­­–r­­–. 1 root root 186444 Oct  9 19:10 sa09
-rw­-r­­–r­­–. 1 root root  53744 Oct 10 22:40 sa10
-rw­-r­­–r­­–. 1 root root 292604 Oct 11 23:30 sa11
[…]
-rw-­r–­­r–­­. 1 root root     67 Sep 30 09:09 sar30

The configuration parameters are explained in the sysstat file itself and they are the same that the command /usr/lib64/sa/sadc understands (man -s8 sadc). But let’s look at SADC_OPTIONS in a bit more detail.

The sa1 and sadc commands (both perform the same task) gather by default most of the statistics we might need (cpu, memory, swapping, paging, etc) but leave out a few that we can explicitly request:

  • INT       → interrupts

    DISK      → disk I/O

    SNMP    → network stats referred in sar as IP,EIP,ICMP,EICMP,TCP,ETCP,UDP

    IPV6       → IPv6 network stats referred in sar as IP6,EIP6,ICMP6,EICMP6,UDP6

    POWER  → power consumption

    XDISK    → extended disk I/O stats down to the partition level

    ALL       → ALL of the above except XDISK

    XALL     → ALL of the above including XDISK

If we need any of the statistic classes above to be gathered, we must include them in the SADC_OPTIONS parameter as shown above. The -D flag used in the example before changes the naming of the binary dump files from saDD to saYYYYMMDD.

Both sa1 and sadc are used to gather stats on an ongoing basis in the background. Sar on the other hand is normally used to gather stats interactively to analyse & resolve performance problems.

Let’s go through some examples and we shall see what sar can do.

Without any flags we get CPU utilisation stats:

root:/etc/sysconfig> sar 2
.
01:24:29 PM    CPU    %user  %nice  %system  %iowait  %steal  %idle
01:24:31 PM    all     3.32   0.00     0.44     3.32    0.00  92.92

With the -u flag we get the same output as above as it is the default flag if none is used:

root:/etc/sysconfig> sar -­u 2
.
01:24:06 PM    CPU  %user  %nice  %system  %iowait  %steal  %idle
01:24:08 PM    all   2.19   0.00     0.38     2.94    0.00  94.49

 

The -B flag gets us paging stats:

root:/etc/sysconfig> sar -­B 2 2
.
01:32:07 PM  pgpgin/s  pgpgout/s  fault/s  majflt/s  pgfree/s  pgscank/s  pgscand/s  pgsteal/s  %vmeff
01:32:09 PM      0.00    1454.00   146.00      0.00    356.00       0.00       0.00       0.00    0.00
01:32:11 PM      0.00    2178.50   166.00      0.00   1090.50       0.00       0.00       0.00    0.00
Average:         0.00    1816.25   156.00      0.00    723.25       0.00       0.00       0.00    0.00

The fields in the example above are:

  • pgpgin/s    → Kbytes paged in per second

    pgpgout/s  → Kbytes paged out per second

    majflt/s      → major page faults (pages not found in memory that require disk I/O) per second

    fault/s        → sum of major + minor page faults (pages in memory but not in the page table that do not require any disk I/O)

    pgfree/s     → number of pages placed in the free list per second

    pgscank/s  → number of pages scanned by the ksawpd daemon per second

    pgscand/s  → number of pages scanned directly per second

    pgsteal/s    → number of pages reclaimed from cache to satisfy memory needs

    %vmeff      → page reclaim success ratio (lower = worse, unless pgscan = 0)

 

The -b flag shows I/O stats:

root:/etc/sysconfig> sar -­b 2 2
.
01:53:17 PM      tps      rtps      wtps    bread/s      bwrtn/s
01:53:19 PM   285.50      0.00    285.50       0.00     13306.00
01:53:21 PM   123.00      0.00    123.00       0.00      5454.00
Average:      204.25      0.00    204.25       0.00      9380.00

The fields in the example above are:

  • tps          → Transfer requests per second issued to the physical device of indeterminate size. Multiple logical requests might be combined into a single physical request.

    rtps         → physical read requests per second

    wtps       → physical write requests per second

    bread/s   → OS blocks or disk sectors (512 bytes) read per second

    bwrtn/s   → OS blocks or disk sectors (512 bytes) written per second

The -d flag shows the activity for each block device:

root:/etc/sysconfig> sar ­-d 2 1 | grep -­v Average:
.
02:03:13 PM        DEV     tps  rd_sec/s   wr_sec/s   avgrq­sz   avgqu­sz   await   svctm   %util
02:03:15 PM      dev8­0  142.50      0.00    4452.00     31.24      2.12   16.36    3.08   43.95
02:03:15 PM     dev8­16  137.00      0.00    4404.00     32.15      2.22   15.93    4.49   61.55
02:03:15 PM    dev9­127  137.50      0.00    4404.00     32.03      0.00    0.00    0.00    0.00
02:03:15 PM    dev253­0    0.00      0.00       0.00      0.00      0.00    0.00    0.00    0.00
02:03:15 PM    dev253­1    2.00      0.00      20.00     10.00      0.00    0.00    0.00    0.00
02:03:15 PM    dev253­2    0.00      0.00       0.00      0.00      0.00    0.00    0.00    0.00
02:03:15 PM    dev253­3    0.00      0.00       0.00      0.00      0.00    0.00    0.00    0.00

The fields in the example above are:

  • tps               → Transfer requests per second issued to the physical device of indeterminate size. Multiple logical requests might be combined into a single physical request.

    rd_sec/s      → OS blocks or disk sectors (512 bytes) read per second

    wr_sec/s      → OS blocks or disk sectors (512 bytes) written per second

    avgrq­sz       → average size in sectors of requests issued

    avgqu­sz      → average queue length of requests

    await           → average time in milliseconds for I/O requests to be serviced (queue+service time)

    svctm         → this figure cannot be trusted and will be removed in future versions

    %util           → percentage utilisation of the device bandwidth

We can use the flag -­j { ID | LABEL | UUID | … } the same way we do with iostat.

The -F flag is used to display space and inode utilisation by filesystems:

root:/etc/sysconfig> sar -­F 1 1 | grep -­v Summary:
.
02:15:54 PM   MBfsfree   MBfsused   %fsused   %ufsused    Ifree   Iused   %Iused   FILESYSTEM
02:15:55 PM       1447       6315     81.36      81.36  5927054  179938     2.95   /dev/mapper/vg­root
02:15:55 PM        466         10      2.01       2.01        0       0     0.00   /dev/sda1
02:15:55 PM      24930       5528     18.15      18.15 31147327   56001     0.18   /dev/mapper/vg­home
02:15:55 PM       2012       1846     47.85      47.85  3944710   16122     0.41   /dev/mapper/vg­var
02:15:55 PM       6194      17869     74.26      79.43  1572851      13     0.00   /dev/mapper/vgdata­sl7ora112
02:15:55 PM       2938       1247     29.79      35.41   280318     242     0.09   /dev/sdb1

The -H flag reports hugepages utilisation:

root:/etc/sysconfig> sar ­-H 1 1
.
02:57:50 PM   kbhugfree   kbhugused   %hugused
02:57:51 PM           0           0       0.00
Average:              0           0       0.00

To get stats on interrupts we should use -­I [ SUM | ALL | XALL | intr1,intr2,… ]:

# the ALL option shows the first 16 !
root:/etc/sysconfig> sar ­-I ALL 5 1 | grep ­v Average:
.
02:59:26 PM      INTR      intr/s
02:59:31 PM         0        0.00
02:59:31 PM         1        0.20
02:59:31 PM         2        0.00
02:59:31 PM         3        0.00
02:59:31 PM         4        0.00
02:59:31 PM         5        0.00
02:59:31 PM         6        0.00
02:59:31 PM         7        0.00
02:59:31 PM         8        0.00
02:59:31 PM         9       42.20
02:59:31 PM        10        0.00
02:59:31 PM        11        0.00
02:59:31 PM        12        0.00
02:59:31 PM        13        0.00
02:59:31 PM        14        0.00
02:59:31 PM        15        0.00
.
# to get figures for one or a few interrupts…
root:/etc/sysconfig> sar -­I 9,10 5 1 | grep ­v Average:
.
03:02:54 PM      INTR      intr/s
03:02:59 PM         9        0.00
03:02:59 PM        10        0.00
.
# or total figures only…
root:/etc/sysconfig> sar ­-I SUM 5 1
.
03:04:00 PM      INTR      intr/s
03:04:05 PM       sum     1101.20
Average:          sum     1101.20
.
# the XALL option will show figures for all interrupts
root:/etc/sysconfig> sar ­-I XALL 5 1 | grep ­v Average:
.
03:14:47 PM      INTR      intr/s
03:14:52 PM         0        0.00
03:14:52 PM         1        7.40
03:14:52 PM         2        0.00
03:14:52 PM         3        0.00
03:14:52 PM         4        0.00
03:14:52 PM         5        0.00
[…]
03:14:52 PM       487        0.00

Warning: If we are going to be executing many iterations of sar we should use the -i flag to ensure that timing stays more or less accurate. Not using it, will cause a small incremental lag for each execution!

We can get power management stats for cpu (CPU), fans (FAN), voltage input (IN), temperature (TEMP) and usb devices (USB) with the -m flag:

root:/etc/sysconfig> sar ­-m CPU 5 1
.
03:32:16 PM    CPU        MHz
03:32:21 PM    all    3301.16
Average:       all    3301.16
.
root:/etc/sysconfig> sar ­-m TEMP 1 1 | grep -­v Average:
.
03:35:33 PM    TEMP     degC    %temp    DEVICE
03:35:34 PM       1    59.00     0.00    acpitz-­virtual-­0
03:35:34 PM       2     0.00     0.00    acpitz-­virtual­-0
03:35:34 PM       3    42.00     0.00    acpitz-­virtual-­0
03:35:34 PM       4     0.00     0.00    nouveau-­pci-­0100
03:35:34 PM       5    60.00    71.43    coretemp­-isa-­0000
03:35:34 PM       6    60.00    71.43    coretemp­-isa­-0000
03:35:34 PM       7    59.00    70.24    coretemp­-isa­-0000
03:35:34 PM       8    59.00    70.24    coretemp­-isa­-0000
.
root:/etc/sysconfig> sar ­-m USB 1 1 | grep -­v Summary
.
03:37:12 PM    BUS   idvendor   idprod   maxpower   manufact         product
03:37:13 PM      1        4f2     b3a6       1000   SunplusIT INC.   HP Truevision HD
03:37:13 PM      3       8087     8008          0
03:37:13 PM      4       8087     8000          0
03:37:13 PM      1       138a       50        200

The -n flag is used to report a variety of network statistics. With the DEV keyword, stats on the network devices are shown:

root:/etc/sysconfig> sar ­-n DEV 5 1 | grep -­v Average:
Linux 4.1.8­100.fc21.x86_64 (envy17)    10/23/2015    _x86_64_   (8 CPU)
.
03:51:27 PM      IFACE   rxpck/s   txpck/s   rxkB/s   txkB/s   rxcmp/s   txcmp/s   rxmcst/s   %ifutil
03:51:32 PM      vnet0      0.00      0.40     0.00     0.02      0.00      0.00       0.00      0.00
03:51:32 PM  virbr0­nic      0.00      0.00     0.00     0.00      0.00      0.00       0.00      0.00
03:51:32 PM       wlo1      0.20      0.20     0.01     0.02      0.00      0.00       0.00      0.00
03:51:32 PM     virbr0      0.00      0.00     0.00     0.00      0.00      0.00       0.00      0.00
03:51:32 PM      vnet1      0.00      0.40     0.00     0.02      0.00      0.00       0.00      0.00
03:51:32 PM       eno1      0.00      0.00     0.00     0.00      0.00      0.00       0.00      0.00
03:51:32 PM         lo     17.80     17.80     1.64     1.64      0.00      0.00       0.00      0.00

  • IFACE        → interface name

    rxpck/s      → packets received per second

    txpck/s       → packets transmitted per second

    rxkB/s        → kbytes received per second

    txkB/s        → kbytes transmittion per second

    rxcmp/s     → number of compressed packets received per second

    txcmp/s     → number of compressed packets transmitted per second

    rxmcst/s    → number of multicast packets received per second

    %ifutil        → percentage of bandwidth utilisation

 

The EDEV keyword is used when we need the error stats of network interfaces:

root:/etc/sysconfig> sar ­-n EDEV 5 1 | grep ­-v Average:
.
05:43:39 PM     IFACE  rxerr/s  txerr/s  coll/s  rxdrop/s  txdrop/s  txcarr/s  rxfram/s  rxfifo/s  txfifo/s
05:43:44 PM     vnet0     0.00     0.00    0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:43:44 PM virbr0­nic     0.00     0.00    0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:43:44 PM      wlo1     0.00     0.00    0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:43:44 PM    virbr0     0.00     0.00    0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:43:44 PM     vnet1     0.00     0.00    0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:43:44 PM      eno1     0.00     0.00    0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:43:44 PM        lo     0.00     0.00    0.00      0.00      0.00      0.00      0.00      0.00      0.00

  • IFACE        → interface name

    rxerr/s        → errorneous packets received per second

    txerr/s        → errorneous packets transmitted per second

    coll/s          → collisions per second

    rxdrop/s     → received packets dropped per second due to full buffers

    txdrop/s     → transmitted packets dropped per second due to full buffers

    txcarr/s      → carrier errors per second while transmitting packets

    rxfram/s     → frame alignment errors per second while receiving packets

    rxfifo/s       → FIFO overruns per second while receiving packets

    txfifo/s       → FIFO overruns per second while transmitting packets

 

The NFS keyword shows NFS client activity stats:

root:/etc/sysconfig> sar -­n NFS 5 1 | grep ­-v Average:
.
05:52:53 PM    call/s   retrans/s   read/s   write/s   access/s   getatt/s
05:52:58 PM     96.00        0.50    84.00      9.00       2.00       1.00

 

The NFSD keyword shows NFS server activity stats:

root:/etc/sysconfig> sar ­-n NFSD 5 1 | grep ­-v Average:
.
05:59:09 PM  scall/s badcall/s packet/s udp/s tcp/s hit/s miss/s sread/s swrite/s saccess/s sgetatt/s
05:59:14 PM     0.00      0.00     0.00  0.00  0.00  0.00   0.00    0.00     0.00      0.00      0.00

 

The SOCK keyword shows stats on IPv4 sockets:

root:/etc/sysconfig> sar -­n SOCK 5 1 | grep -­v Average:
.
06:02:19 PM  totsck  tcpsck  udpsck  rawsck  ip­frag  tcp­tw
06:02:24 PM     872      22      10       0       0     11

  • totsck     → total number of sockets in use

    tcpsck    → number of tcp sockets in use

    udpsck   → number of udp sockets in use

    rawsck   → number of raw sockets in use

    ip­-frag     → number of IP fragments in queue

    tcp-­tw      → number of sockets in TIME_WAIT state (localhost has closed connection but socket is kept open a bit longer waiting for delayed packets)

 

The IP keyword shows stats on IPv4 traffic:

root:/etc/sysconfig> sar -­n IP 5 1 | grep -­v Average:
.
06:12:23 PM   irec/s   fwddgm/s   idel/s   orq/s   asmrq/s   asmok/s   fragok/s   fragcrt/s
06:12:28 PM    16.60       0.00    16.60   16.60      0.00      0.00       0.00        0.00

irec/s         → number of input datagrams received from interfaces per second, including those received in error [ipInReceives].

fwddgm/s  → number of input datagrams per second, for which this entity was not their final
IP destination, as a result of which an attempt was made to find a route to forward them to that final destination [ipForwDatagrams].

idel/s         → number of input datagrams successfully delivered per second to IP user­ protocols (including ICMP) [ipInDelivers].

orq/s          → number of IP datagrams which local IP user­protocols (including ICMP) supplied per second to IP in requests for transmission [ipOutRequests]. Note that this counter does not include any datagrams counted in fwddgm/s.

asmrq/s     → number of IP fragments received per second which needed to be reassembled at this entity [ipReasmReqds].

asmok/s     → number of IP datagrams successfully re­assembled per second [ipReasmOKs].

fragok/s     → number of IP datagrams that have been successfully fragmented at this entity per second [ipFragOKs].

fragcrt/s    → number of IP datagram fragments that have been generated per second as a result of fragmentation at this entity [ipFragCreates].

 

The TCP keyword shows stats on TCP traffic:

root:/etc/sysconfig> sar -­n TCP 5 1 | grep ­-v Average:
.
07:10:55 PM    active/s   passive/s   iseg/s   oseg/s
07:11:00 PM        1.00        0.00    26.40    28.00

 

The UDP keyword shows stats on UDP traffic:

root:/etc/sysconfig> sar -­n UDP 5 1 | grep ­-v Average:
.
07:11:47 PM   idgm/s   odgm/s   noport/s   idgmerr/s
07:11:52 PM     0.00     0.00       0.00        0.00

 

The ICMP keyword shows stats on ICMP traffic:

root:/etc/sysconfig> sar -­n ICMP 5 1 | grep ­-v Average:
07:12:44 PM  imsg/s omsg/s iech/s iechr/s oech/s oechr/s itm/s itmr/s otm/s otmr/s iadrmk/s iadrmkr/s oadrmk/s oadrmkr/s
07:12:49 PM    0.00   0.00   0.00    0.00   0.00    0.00  0.00   0.00  0.00   0.00     0.00      0.00     0.00          0.00

  • imsg/s       -> number of ICMP messages which the entity received per second [icmpInMsgs]. Note that this counter includes all those counted by ierr/s.

  • omsg/s       -> number of ICMP messages which this entity attempted to send per second [icmpOutMsgs]. Note that this counter includes all those counted by oerr/s.

  • iech/s         -> number of ICMP Echo (request) messages received per second [icmpInEchos].

  • iechr/s        -> number of ICMP Echo Reply messages received per second [icmpInEchoReps].

  • oech/s        -> number of ICMP Echo (request) messages sent per second [icmpOutEchos].

  • oechr/s       -> number of ICMP Echo Reply messages sent per second [icmpOutEchoReps].

  • itm/s            -> number of ICMP Timestamp (request) messages received per second [icmpInTimestamps].

  • itmr/s           -> number of ICMP Timestamp Reply messages received per second [icmpInTimestampReps].

  • otm/s           -> number of ICMP Timestamp (request) messages sent per second [icmpOutTimestamps].

  • otmr/s          -> number of ICMP Timestamp Reply messages sent per second [icmpOutTimestampReps].

  • iadrmk/s      -> number of ICMP Address Mask Request messages received per second [icmpInAddrMasks].

  • iadrmkr/s     -> number of ICMP Address Mask Reply messages received per second [icmpInAddrMaskReps].

  • oadrmk/s     -> number of ICMP Address Mask Request messages sent per second [icmpOutAddrMasks].

  • oadrmkr/s    -> number of ICMP Address Mask Reply messages sent per second [icmpOutAddrMaskReps].

 

The SOCK6 keyword shows stats on IPv6 sockets:

root:/etc/sysconfig> sar ­-n SOCK6 5 1 | grep ­-v Average:
.
07:13:33 PM    tcp6sck    udp6sck    raw6sck    ip6­frag
07:13:38 PM          2          5          1          0

  • tcp6sck      -> number of TCPv6 sockets currently in use

  • udp6sck     -> number of UDPv6 sockets currently in use

  • raw6sck     -> number of RAWv6 sockets currently in use

  • ip6­frag        -> number of IPv6 fragments currently in use

 

The IP6 keyword shows stats on IPv6 traffic:

root:/etc/sysconfig> sar -­n IP6 5 1 | grep ­-v Average:
.
06:21:13 PM irec6/s fwddgm6/s idel6/s orq6/s asmrq6/s asmok6/s imcpck6/s omcpck6/s fragok6/s fragcr6/s
06:21:18 PM    0.00      0.00    0.00   0.00     0.00     0.00      0.00      0.00       0.00     0.00

irec6/s         -> number of input datagrams received from interfaces per second, including those received in error [ipv6IfStatsInReceives].

fwddgm6/s  -> number of output datagrams per second which this entity received and forwarded to their final destinations [ipv6IfStatsOutForwDatagrams].

idel6/s          -> number of datagrams successfully delivered per second to IPv6 user­protocols (including ICMP) [ipv6IfStatsInDelivers].

orq6/s          -> number of IPv6 datagrams which local IPv6 user­protocols (including ICMP) supplied per second to IPv6 in requests for transmission [ipv6IfStatsOutRequests]. Note that this counter does not include any datagrams counted in fwddgm6/s.

asmrq6/s     -> number of IPv6 fragments received per second which needed to be reassembled at this interface [ipv6IfStatsReasmReqds].

asmok6/s     -> number of IPv6 datagrams successfully reassembled per second [ipv6IfStatsReasmOKs].

imcpck6/s    -> number of multicast packets received per second by the interface [ipv6IfStatsInMcastPkts].

omcpck6/s   -> number of multicast packets transmitted per second by the interface [ipv6IfStatsOutMcastPkts].

fragok6/s      -> number of IPv6 datagrams that have been successfully fragmented at this output interface per second [ipv6IfStatsOutFragOKs].

fragcr6/s      -> number of output datagram fragments that have been generated per second as a result of fragmentation at this output interface [ipv6IfStatsOutFragCreates].

 

The UDP6 keyword shows stats on UDP6 traffic:

root:/etc/sysconfig> sar -­n UDP6 5 1 | grep -­v Average:
.
07:14:39 PM  idgm6/s  odgm6/s  noport6/s  idgmer6/s
07:14:44 PM     0.00     0.00       0.00       0.00

idgm6/s   -> number of UDP datagrams delivered per second to UDP users [udpInDatagrams]

odgm6/s  -> number of UDP datagrams sent per second from this entity [udpOutDatagrams]

noport6/s -> number of received UDP datagrams per second for which there was no application at the destination port [udpNoPorts]

idgmer6/s -> number of received UDP datagrams per second that could not be delivered for reasons other than the lack of an application at the destination port [udpInErrors]

 

The ICMP6 keyword shows stats on ICMP6 traffic:

root:/etc/sysconfig> sar -­n ICMP6 5 1 | grep -­v Average:
.
07:15:36 PM imsg6/s omsg6/s iech6/s iechr6/s oechr6/s igmbq6/s igmbr6/s ogmbr6/s igmbrd6/s ogmbrd6/s irtsol6/s ortsol6/s irtad6/s inbsol6/s onbsol6/s inbad6/s onbad6/s
07:15:41 PM 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

imsg6/s      number of ICMP messages received by the interface per second which includes all those counted by ierr6/s [ipv6IfIcmpInMsgs].

omsg6/s     number of ICMP messages which this interface attempted to send per second [ipv6IfIcmpOutMsgs].

iech6/s        number of ICMP Echo (request) messages received by the interface per second [ipv6IfIcmpInEchos].

iechr6/s       number of ICMP Echo Reply messages received by the interface per second [ipv6IfIcmpInEchoReplies].

oechr6/s      number of ICMP Echo Reply messages sent by the interface per second [ipv6IfIcmpOutEchoReplies].

igmbq6/s     number of ICMPv6 Group Membership Query messages received by the interface per second [ipv6IfIcmpInGroupMembQueries].

igmbr6/s      number of ICMPv6 Group Membership Response messages received by the interface per second [ipv6IfIcmpInGroupMembResponses].

ogmbr6/s     number of ICMPv6 Group Membership Response messages sent per second [ipv6IfIcmpOutGroupMembResponses].

igmbrd6/s    number of ICMPv6 Group Membership Reduction messages received by the interface per second [ipv6IfIcmpInGroupMembReductions].

ogmbrd6/s  number of ICMPv6 Group Membership Reduction messages sent per second [ipv6IfIcmpOutGroupMembReductions].

irtsol6/s      number of ICMP Router Solicit messages received by the interface per second [ipv6IfIcmpInRouterSolicits].

ortsol6/s     number of ICMP Router Solicitation messages sent by the interface per second [ipv6IfIcmpOutRouterSolicits].

irtad6/s       number of ICMP Router Advertisement messages received by the interface per second [ipv6IfIcmpInRouterAdvertisements].

inbsol6/s    number of ICMP Neighbor Solicit messages received by the interface per second [ipv6IfIcmpInNeighborSolicits].

onbsol6/s   number of ICMP Neighbor Solicitation messages sent by the interface per second [ipv6IfIcmpOutNeighborSolicits].

inbad6/s     number of ICMP Neighbor Advertisement messages received by the interface per second [ipv6IfIcmpInNeighborAdvertisements].

onbad6/s    number of ICMP Neighbor Advertisement messages sent by the interface per second [ipv6IfIcmpOutNeighborAdvertisements].

 

The EIP keyword shows IPv4 error stats:

root:/etc/sysconfig> sar -­n EIP 5 1 | grep -­v Average:
.
06:31:03 PM  ihdrerr/s  iadrerr/s  iukwnpr/s  idisc/s  odisc/s  onort/s  asmf/s  fragf/s
06:31:08 PM       0.00       0.00       0.00     0.00     0.00     0.00    0.00     0.00

ihdrerr/s   number of input datagrams discarded per second due to errors in their IP headers, including bad checksums, version number mismatch, other format errors, time-­to-­live exceeded, errors discovered in processing their IP options, etc. [ipInHdrErrors]

iadrerr/s   number of input datagrams discarded per second because the IP address in their IP header’s destination field was not a valid address to be received at this entity. This count includes nvalid addresses (e.g., 0.0.0.0) and addresses of unsupported  Classes (e.g., Class E). For entities hich are not IP routers and therefore do not forward datagrams, this counter includes datagrams scarded because the destination address was not a local address [ipInAddrErrors].

iukwnpr/s   number of locally­addressed datagrams received successfully but discarded per econd because of an unknown or unsupported protocol [ipInUnknownProtos].

idisc/s   number of input IP datagrams per second for which no problems were encountered to prevent their continued processing, but which were discarded (e.g., for lack of buffer space) [ipInDiscards]. Note that this counter does not include any datagrams discarded while awaiting re­assembly.

odisc/s   number of output IP datagrams per second for which no problem was encountered to prevent their transmission to their destination, but which were discarded (e.g., for lack of buffer space) ipOutDiscards]. Note that this counter would include datagrams counted in fwddgm/s if any such packets met this (discretionary) discard criterion.

onort/s   number of IP datagrams discarded per second because no route could be found to transmit them to their destination [ipOutNoRoutes]. Note that this counter includes any packets counted in wddgm/s which meet this ‘no­route’ criterion. Note that this includes any datagrams which a host cannot route because all of its default routers are down.

asmf/s   number of failures detected per second by the IP re­assembly algorithm (for whatever reason: timed out, errors, etc) [ipReasmFails]. Note that this is not necessarily a count of discarded IP fragments since some algorithms can lose track of the number of fragments by combining them as they are received.

fragf/s   number of IP datagrams that have been discarded per second because they needed to be fragmented at this entity but could not be, e.g., because their Don’t Fragment flag was set [ipFragFails].

 

The ETCP keyword shows TCP error stats:

root:/etc/sysconfig> sar -­n ETCP 5 1 | grep ­-v Average:
.
07:16:43 PM  atmptf/s  estres/s  retrans/s  isegerr/s  orsts/s
07:16:48 PM      0.00      0.00       0.00       0.00     0.00

atmptf/s    number of times per second TCP connections have made a direct transition to the CLOSED state from either the SYN­SENT state or the SYN­RCVD state, plus the number of times per second TCP connections have made a direct transition to the LISTEN state from the SYN­RCVD state [tcpAttemptFails].

estres/s    number of times per second TCP connections have made a direct transition to the CLOSED state from either the ESTABLISHED state or the CLOSE­WAIT state [tcpEstabResets].

retrans/s   number of segments retransmitted per second ­ that is, the number of TCP segments transmitted containing one or more previously transmitted octets [tcpRetransSegs].

isegerr/s   number of segments received in error (e.g., bad TCP checksums) per second [tcpInErrs].

orsts/s      number of TCP segments sent per second containing the RST flag [tcpOutRsts].

 

The EICMP keyword shows ICMP error stats:

root:/etc/sysconfig> sar ­-n EICMP 5 1 | grep -­v Average:
.
07:17:53 PM ierr/s oerr/s idstunr/s odstunr/s itmex/s otmex/s iparmpb/s oparmpb/s isrcq/s osrcq/s iredir/s oredir/s
07:17:58 PM  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00

ierr/s    number of ICMP messages per second which the entity received but determined as having ICMP­specific errors (bad ICMP checksums, bad length,etc.) [icmpInErrors].

oerr/s   number of ICMP messages per second which this entity did not send due to problems discovered within ICMP such as a lack of buffers [icmpOutErrors].

idstunr/s   number of ICMP Destination Unreachable messages received per second [icmpInDestUnreachs].

odstunr/s   number of ICMP Destination Unreachable messages sent per second [icmpOutDestUnreachs].

itmex/s   number of ICMP Time Exceeded messages received per second [icmpInTimeExcds].

otmex/s   number of ICMP Time Exceeded messages sent per second [icmpOutTimeExcds].

iparmpb/s   number of ICMP Parameter Problem messages received per second [icmpInParmProbs].

oparmpb/s   number of ICMP Parameter Problem messages sent per second [icmpOutParmProbs].

isrcq/s   number of ICMP Source Quench messages received per second [icmpInSrcQuenchs].

osrcq/s   number of ICMP Source Quench messages sent per second [icmpOutSrcQuenchs].

iredir/s   number of ICMP Redirect messages received per second [icmpInRedirects].

oredir/s   number of ICMP Redirect messages sent per second [icmpOutRedirects].

 

The EIP6 keyword shows IPv6 error stats:

root:/etc/sysconfig> sar ­-n EIP6 5 1 | grep -­v Average:
.
06:41:43 PM ihdrer6/s iadrer6/s iukwnp6/s i2big6/s idisc6/s odisc6/s inort6/s onort6/s asmf6/s fragf6/s itrpck6/s
06:41:48 PM      0.00      0.00      0.00     0.00     0.00     0.00     0.00     0.00    0.00    0.00       0.00

ihdrer6/s   number of input datagrams discarded per second due to errors in their IPv6 headers, including version number mismatch, other format errors, hop count exceeded, errors discovered in processing their IPv6 options, etc. [ipv6IfStatsInHdrErrors]

iadrer6/s   number of input datagrams discarded per second because the IPv6 address in their IPv6 header’s destination field was not a valid address to be received at this entity. This count includes invalid addresses (e.g., ::0) and unsupported addresses (e.g., addresses with unallocated prefixes). For entities which are not IPv6 routers and therefore do not forward datagrams, this counter includes datagrams discarded because the destination address was not a local address [ipv6IfStatsInAddrErrors].

iukwnp6/s   number of locally­addressed datagrams received successfully but discarded per second because of an unknown or unsupported protocol [ipv6IfStatsInUnknownProtos].

i2big6/s   number of input datagrams that could not be forwarded per second because their size exceeded the link MTU of outgoing interface [ipv6IfStatsInTooBigErrors].

idisc6/s   number of input IPv6 datagrams per second for which no problems were encountered to prevent their continued processing, but which were discarded (e.g., for lack of buffer space) [ipv6IfStatsInDiscards]. Note that this counter does not include any datagrams discarded while awaiting re­assembly.

odisc6/s   number of output IPv6 datagrams per second for which no problem was encountered to prevent their transmission to their destination, but which were discarded (e.g., for lack of buffer space) [ipv6IfStatsOutDiscards]. Note that this counter would include datagrams counted in fwddgm6/s if any such packets met this (discretionary) discard criterion.

inort6/s   number of input datagrams discarded per second because no route could be found to transmit them to their destination [ipv6IfStatsInNoRoutes].

onort6/s   number of locally generated IP datagrams discarded per second because no route could be found to transmit them to their destination [unknown formal SNMP name].

asmf6/s   number of failures detected per second by the IPv6 re­assembly algorithm (for whatever reason: timed out, errors, etc.) [ipv6IfStatsReasmFails]. Note that this is not necessarily a count of discarded IPv6 fragments since some algorithms can lose track of the number of fragments by combining them as they are received.

fragf6/s   number of IPv6 datagrams that have been discarded per second because they needed to be fragmented at this output interface but could not be [ipv6IfStatsOutFragFails].

itrpck6/s   number of input datagrams discarded per second because datagram frame didn’t carry enough data [ipv6IfStatsInTruncatedPkts].

 

The EICMP6 keyword shows ICMP6 error stats:

root:/etc/sysconfig> sar ­-n EICMP6 5 1 | grep ­-v Average:
.
07:18:40 PM ierr6/s idtunr6/s odtunr6/s itmex6/s otmex6/s iprmpb6/s oprmpb6/s iredir6/s oredir6/s ipck2b6/s
opck2b6/s
07:18:45 PM    0.00      0.00      0.00     0.00     0.00      0.00      0.00      0.00      0.00      0.00
0.00

ierr6/s   number of ICMP messages per second which the interface received but determined as having ICMP­specific errors (bad ICMP checksums, bad length, etc.) [ipv6IfIcmpInErrors]

idtunr6/s   number of ICMP Destination Unreachable messages received by the interface per second [ipv6IfIcmpInDestUnreachs].

odtunr6/s   number of ICMP Destination Unreachable messages sent by the interface per second [ipv6IfIcmpOutDestUnreachs].

itmex6/s   number of ICMP Time Exceeded messages received by the interface per second [ipv6IfIcmpInTimeExcds].

otmex6/s   number of ICMP Time Exceeded messages sent by the interface per second [ipv6IfIcmpOutTimeExcds].

iprmpb6/s   number of ICMP Parameter Problem messages received by the interface per second [ipv6IfIcmpInParmProblems].

oprmpb6/s   number of ICMP Parameter Problem messages sent by the interface per second [ipv6IfIcmpOutParmProblems].

iredir6/s   number of Redirect messages received by the interface per second [ipv6IfIcmpInRedirects].

oredir6/s   number of Redirect messages sent by the interface by second [ipv6IfIcmpOutRedirects].

ipck2b6/s   number of ICMP Packet Too Big messages received by the interface per second [ipv6IfIcmpInPktTooBigs].

opck2b6/s   number of ICMP Packet Too Big messages sent by the interface per second [ipv6IfIcmpOutPktTooBigs].

 

We can combine more than one keyword plus other flags to extract various statistics at the same time:

root:/etc/sysconfig> sar ­-r -­m CPU ­-b ­-n ICMP,ICMP6,EICMP6 5 1

 

The -q flag shows CPU queues and load averages:

root:/etc/sysconfig> sar -­q 5 1 | grep ­-v Average:
11:12:15 PM runq­sz plist­sz ldavg­1 ldavg­5 ldavg­15 blocked
11:12:20 PM 0 629 0.90 1.02 1.11 0

runq­sz   → run queue (number of threads waiting for run time)

plist­sz    → number of threads in any state currently in the system

ldavg­1    → load average for the past minute

ldavg­5    → load average for the past 5 minutes

ldavg­15  → load average for the past 15 minutes

blocked  → number of threads waiting for I/O to complete

The -R flag shows memory activity stats:

root:/etc/sysconfig> sar -­R 5 1 | grep -­v Average:
.
11:14:27 PM   frmpg/s   bufpg/s   campg/s
11:14:32 PM    487.40     0.00     ­519.40

frmpg/s    → number of pages (4KB) freed per second

bufpg/s    → number of pages added to buffers per second

campg/s   → number of pages added to cache per second

 

 

The -r flag shows memory utilisation stats:

root:/etc/sysconfig> sar ­-r 5 1 | grep -­v Average:
.
11:15:28 PM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty
11:15:33 PM   3439828  12762160    78.77    137400  8390228  6632108   32.83  8151528 3204224    3136

kbmemfree    → free memory in Kbytes

kbmemused   → used memory in Kbytes

%memused    → percentage of total memory used

kbbuffers       → I/O buffers size in Kbytes

kbcached       → file system cache size in Kbytes

kbcommit      → memory needed for current workload in Kbytes

%commit       → percentage of memory needed for current workload

kbactive         → active memory in Kbytes

kbinact           → inactive memory in Kbytes (reclaimable)

kbdirty           → dirty memory in Kbytes that is due to be written to disk

 

The -S flag shows memory utilisation stats:

root:/etc/sysconfig> sar -­S 5 1 | grep -­v Average:
.
11:15:51 PM   kbswpfree   kbswpused   %swpused   kbswpcad   %swpcad
11:15:56 PM     3999740           0       0.00          0      0.00

kbswpfree    → free swap space in Kbytes

kbswpused  → used swap space in Kbytes

%swpused   → percentage of swap space used

kbswpcad    → cached swap memory in Kbytes (memory both in RAM and swap)

%swpcad     → percentage of cached swap memory

 

The -v flag shows memory utilisation stats:

root:/etc/sysconfig> sar ­-v 5 1 | grep ­-v Average:
.
11:18:57 PM    dentunusd    file­nr    inode­nr    pty­nr
11:19:02 PM      1479181     12114     558983        4

dentunusd   → number of unused cache entries in the directory cache

file­nr            → number of filehandles in use system­wide

inode­nr       → number of inode handlers in use system­wide

pty­nr           → number of pseudo­terminals in use system­wid

 

The -W flag shows the number of pages swapped-in and swapped-out per second:

root:/etc/sysconfig> sar ­-W 5 1 | grep -­v Average:
.
11:19:22 PM    pswpin/s    pswpout/s
11:19:27 PM        0.00         0.00

 

The -w flag shows the number of threads created and context switches per second:

root:/etc/sysconfig> sar -­w 5 1 | grep -­v Average:
.
11:19:50 PM    proc/s    cswch/s
11:19:55 PM      1.00    1809.40

We can use the flag -A as an alias of “ -bBdFHqrRSuvwWy -I SUM -I XALL -m ALL -n ALL -u ALL -P ALL ”. When -A is used though the output tends to be sent to an output (-o) file rather than the terminal (too much data!):

root:/etc/sysconfig> sar ­-A -­o /tmp/all.stats

We can read files with sar binary data with the -f flag:

root:/etc/sysconfig> sar ­-f /var/log/sa/sa15 ­-d ­-s 15:30:00 ­-e 16:00:00 

The example above extracts the device stats from the given file for the time lapse 15:30 – 16:00. We can use the start (-s) and end (-e) flags also when writing to a file.

<< iostat                      netstat >>