atop is in functionality and aim very similar to top and htop. It can be used interactively and in the background dumping gathered stats into binary files. However, as there are other tools (e.g. sar) that do the data gathering & dumping really well, we shall focus only in the interactive aspect of atop which is the one most people use.

As it the case with htop, atop needs to be manually installed as it does not come by default:

# dnf -y install atop

To launch the interactive ncurses GUI we just execute atop in the command line:

In the top frame, from left-to-right we can see:

PRC | sys 0.38s | user 2.26s
Number of seconds spent on sys and user mode by all the cores in the last time interval. If the interval is kept to the default 10 seconds, 0.38 + 2.26 seconds the cores were busy and 7.36 they were waiting for I/O to complete or idle.

#proc 369 | #trun 2 | #tslpi 700 | #tslpu 0 | #zombie 2 | clones 19 | | #exit 16
Number of current processes, threads running or runnable, threads in interruptible sleep,
threads in uninterruptible sleep, zombie processes and processes that exited in the last
interval.

CPU | sys 5% | user 26% | irq 0% | | idle 74% | wait 24% | | steal 0% | guest 0% | curf 3.25GHz | curscal 95%
Percentage of all CPU time spent on system, user, interrupt, idle, I/O wait, steal & guest modes.
The steal mode applies only to VMs as it implies the time stolen by other VMs running on the same system.
The guest applies only to the VM host and it shows the time used by its VM guests.
The curf and curscal values shows the average frequency and scaling of all the cores. If the CPU architecture allows it, the frequency of cores is throttled down by many OSes where the load is low in order to save power. The CPU utilisation figures already take this into account. For example: 50% utilisation of a core at 50% max frequency = 25% effective utilisation.

cpu | sys 2% | user 13% | irq 0% | | idle 67% | cpu004 w 17% | | steal 0% | guest 0% | curf 3.27GHz | curscal 96%
All the lines starting with “cpu” in lowercase show the stats for a particular core.

CPL | avg1 1.41 | avg5 1.24 | | avg15 1.22 | | csw 16157 | | intr 12268 | | | numcpu 8
Average load in the last 1, 5 and 15 minutes, number of context switches, number of interrupts and number of cores available.

MEM | tot 15.5G | free 3.3G | cache 8.6G | dirty 5.1M | buff 137.8M | slab 408.1M | | slrec 288.3M | shmem 460.9M | shrss 21.3M | vmbal 0.0M | hptot 0.0M | hpuse 0.0M
Total amount of memory available, free memory, cached memory, dirty memory that is due to be written to disk, I/O buffers size, kernel memory size, kernel reclaimable memory, shared memory, resident shared memory, swapped memory, Vmware’s baloon driver memory, hugepages total memory and hugepages used memory.

SWP | tot 3.8G | free 3.8G | | | | | | | | | | | vmcom 6.6G | vmlim 11.5G | |
Total swap space available, free swap space, virtual memory commited space and virtual memory max limit (default vlim = swap available + 50% of RAM)

PAG | scan 0 | stall 0 | swin 0 | swout 0
This line is not shown in the screenshot above because no paging occurred. But the figures are for number of pages scanned and number of stalls. The kernel scans memory looking for pages it can reclaim when the free memory falls below a certain threshold (stall event).
Swin and swout are the number of pages pulled from swap into memory (former) and number of pages pushed from memory onto swap (latter).

LVM/MDD/DSK | sdb | busy 71% || read 0 | write 1381 || KiB/r 0 | KiB/w 15 ||
MBr/s 0.00 | MBw/s 2.04 || avq 3.59 | avio 5.11 ms ||
Name of logical volume, software RAID volume or disk, percentage of time the device was busy handling I/O, read & write requests, Kbytes and Mbytes read/written, average I/O queue size and average service time per I/O request (seek + latency + data transfer).

NET | transport | tcpi 4987 | tcpo 5018 | | udpi 1301 | udpo 1164 | tcpao 362 | tcppo 0 | | tcprs 26 | tcpie 11 | tcpor 22 | udpnp 3 | | udpie 0 |
Number of TCP packets received and transmitted, UDP packets received and transmitted, TCP connections active and passive, TCP retransmissions, TCP input and output errors, UDP no ports and input errors. Figures shown are IPv4 + IPv6.

NET | network | ipi 6327 | | ipo 6211 | ipfrw 0 | | deliv 6320 | | | | | | icmpi 0 | icmpo 0
Number of IP packets received, forwarded and sent, and number of ICMP packets received and  sent.

NET | wlo1 0% | pcki 6178 | | pcko 6114 | si 20 Kbps | so 5 Kbps | | coll 0 | mlti  0 | | erri 0 | erro 0 | drpi 0 | | drpo 0 |
Interface name, input & output packets, bits received & sent, collisions, received multicast packets, input & output errors and dropped input & output packets.

 

The lower pane shows information related to processes & threads. By default the generic pane is shown, but if it isn’t and we want to go back to it we can do so pressing the g key:

 PID   TID RUID EUID THR SYSCPU USRCPU VGROW RGROW RDDSK WRDSK ST EXC S CPUNR CPU CMD
8435 ­   –  marc marc   9  0.13s  1.02s    0K   20K    0K 9104K­ —   – S    3  23% insync
2338 ­   –  marc marc   7  0.02s  0.08s    0K    0K    0K    0K ­­­–   – S    1   2% gnome
1535 ­   –  root root   4  0.04s  0.04s    0K    0K    0K    0K ­­­–   – S    0   2% Xorg.bin
2681 ­   –  marc marc   4  0.00s  0.04s    0K    0K    0K    0K —   – S    4   1% gnome­

• PID            -> process ID
  TID            -> thread ID
  RUID         -> real user ID
  EUID         -> effective user ID
  THR          -> number of threads
  SYSCPU   -> system cpu time used
  USRCPU   -> user cpu time used
  VGROW    -> virtual memory growth in last interval
  RGROW    -> resident memory growth in the last interval
  RDDSK     -> physical read I/O of this process and all its children
  WRDSK     -> physical written I/O of this process and all its children
  ST             -> status of the process (N=new process, E=exit voluntarily, S=signal terminated,
                                                           C=signal terminated + core dump produced)
  EXC          -> exit code of process (usually 0  →  no problem, >0  →  problem somewhere)
  S               -> status (R=running/runnable, S=interruptible sleep, D=uninterruptible sleep,
                                   Z=zombie, T=stopped or being traced, W=swapping, E=exited)
  CPUNR     -> cpu ID process is or has been running on
  CPU          -> percentage of cpu used
  CMD         -> command or binary being executed

 

We can get the process pane by pressing the p key:

NPROCS SYSCPU USRCPU  VSIZE  RSIZE PSIZE SWAPSZ  RDDSK  WRDSK RNET SNET CPU CMD
1   50.47s  6m05s   2.7G 209.7M    0K     0K     8K   3.3G    0    0 25% insync
1    1.44s 15.41s   2.2G 751.7M    0K     0K     0K   120K    0    0  1% gnome­shell
1    0.78s 15.02s   1.7G 360.0M    0K     0K 19572K 96532K    0    0  1% soffice.bin
8    2.45s 12.43s   7.6G   1.2G    0K     0K 29480K 227.5M    0    0  1% chrome
1    6.61s  7.01s 537.6M 92184K    0K     0K     0K     0K    0    0  1% Xorg.bin

Most stats in this layout are cumulative since process startup and not per interval!

NPROCS   → number of child processes forked
SYSCPU   → system cpu time used
USRCPU   → user cpu time used
VSIZE        → virtual memory used
RSIZE        → resident memory used
PSIZE        → approximate unshared resident memory
SWAPSZ   → swap space used
RDDSK     → physical I/O of this process and all its children
WRDSK    → physical I/O of this process and all its children
RNET       → number of TCP + UDP packets received (kernel module netatop is required)
SNET       → number of TCP + UDP packets transmitted (kernel module netatop is required)
CPU         → percentage of CPU used by this process
CMD         → command or binary being executed

 

We can get the memory pane by pressing the m key:

PID   TID MINFLT MAJFLT VSTEXT VSLIBS  VDATA VSTACK VSIZE  RSIZE PSIZE VGROW RGROW SWAPSZ RUID EUID MEM CMD
2338   – ­     0      0    16K 68424K   1.5G   184K  2.2G 752.2M    0K    0K    0K     0K marc marc  5% gnome­shell
5001   – ­   305      0 90844K 37960K 858.3M   136K  1.5G 518.2M    0K    0K   72K     0K marc marc  3% chrome
11022   – ­    14      0 90844K 50584K 701.6M   136K  1.2G 241.2M    0K    0K    0K     0K marc marc  2% chrome
22465   – ­     0      0 31268K 104.0M 471.2M   136K  1.2G 213.0M    0K    0K    0K     0K marc marc  1% spotify

PID          → process ID
TID          → thread ID
MINFLT   → number of minor page faults
MAJFLT  → number of major page faults
VSTEXT  → virtual shared memory used for text segments
VSLIBS   → virtual shared memory used for library segments
VDATA    → virtual memory used for data
VSTACK → virtual memory used for stack
VSIZE     → total virtual memory size
RSIZE     → resident memory size
PSIZE     → process memory size (you need to press R to get this figure!)
VGROW  → virtual memory growth in last interval
RGROW  → resident memory growth in last interval
SWAPSZ → swap memory used
RUID       → real user ID
EUID       → effective user ID
MEM       → percentage of total memory used by this process
CMD       → command or binary being executed

 

We can get the device pane by pressing the d key:

  PID   TID   RDDSK     WRDSK    WCANCL    DSK   CMD
8435     – ­     0K    10436K        0K    91%   insync
2527     – ­     0K      504K       12K     4%   tracker­store
2515     – ­   460K        0K        0K     4%   tracker­extrac
5001     – ­     0K       84K        4K     1%   chrome

PID             → process ID
TID             → thread ID
RDDSK      → physical I/O of this process and all its children
WRDSK     → physical I/O of this process and all its children
WCANCL   → physical I/O written and withdrawn
DSK           → percentage of disk bandwidth used
CMD          → command or binary being executed

 

We can get the network pane by pressing the n key if the netatop kernel module is installed:

  PID TID TCPRCV TCPRASZ TCPSND TCPSASZ UDPRCV UDPRASZ UDPSND UDPSASZ  BANDWI  BANDWO  NET CMD
5001 ­  –     48     258     65     264     13     169     15     201 23 Kbps 32 Kbps 100% chrome
8435   -­      0       0      0       0      0       0      0       0  0 Kbps  0 Kbps   0% insync
1535   – ­     0       0      0       0      0       0      0       0  0 Kbps  0 Kbps   0% Xorg.bin
5148   – ­     0       0      0       0      0       0      0       0  0 Kbps  0 Kbps   0% chrome
2338   – ­     0       0      0       0      0       0      0       0  0 Kbps  0 Kbps   0% gnome­she

PID          → process ID
TID          → thread ID
ENVID     → field not used
TRUN      → threads running or runnable
TSLPI      → threads in interruptible sleep
TSLPU    → threads in uninterruptible sleep
POLI       → scheduling policy: normal, batch, idle, fifo and round­robin
NICE       → niceness level from ­20 (highest) to +19 (lowest)
PRI          → priority level from 0 (highest) to 139 (lowest)
RPTR      → real­time priority: 0 for timesharing classes (normal/batch/idle) and 1 (lowest) to 99
(highest) for process in realtime class (round­robin/fifo)
CPUNR   → cpu ID process is or has been running on
ST           → status of the process (N=new process, E=exit voluntarily, S=signal terminated,
C=signal terminated + core dump produced)
EXC        → exit code (only for terminated processes)
S             → status (R=running/runnable, S=interruptible sleep, D=uninterruptible sleep,
Z=zombie, T=stopped or being traced, W=swapping, E=exited)
CPU        → percentage of CPU time used
CMD        → command or binary being executed

 

We can get the wide command line pane by pressing the c key:

  PID TID  S CPU COMMAND­-LINE (horizontal scroll with <­ and ­> keys)
8435   – ­ S 28% ./insync start
7913   – ­ R  4% atop
22465   – ­ S  1% /usr/lib64/spotify­client/spotify
2338   -­  S  1% /usr/bin/gnome­shell
2681   – ­ S  0% /usr/libexec/gnome­terminal­server
2285   – ­ S  0% /usr/bin/pulseaudio ­­start
786   – ­ S  0% irq/32­iwlwifi
11022   – ­ S  0% /opt/google/chrome/chrome ­­type=renderer ­­lang=en­US –force­fieldtrials=*AffiliationBasedMatchi
1535   – ­ S  0% /usr/libexec/Xorg.bin :0 ­background none ­noreset ­verbose 3 ­logfile /dev/null ­auth

PID    → process ID
TID    → thread ID
S       → status (R=running/runnable, S=interruptible sleep, D=uninterruptible sleep,
Z=zombie, T=stopped or being traced, W=swapping, E=exited)
CPU   → percentage of CPU time used
CMD  → command or binary being executed

 

We can get the thread pane by pressing the y key:

  PID   TID S CPU COMMAND-­LINE (horizontal scroll with <­ and ­> keys)
8435     – ­S 28% ./insync start
8435  8435 S  2% insync
8435  8436 D 25% insync
8435  8450 S  0% dconf worker
8435  8451 S  0% gdbus
8435  8455 S  0% insync
8435  8456 S  0% insync
8435  8463 S  0% QThread
8435  8465 S  0% insync
8435  8466 S  0% QThread
7913     -­ R  4% atop
22465     -­ S  1% /usr/lib64/spotify­client/spotify
22465 22497 S  0% Network Thread
22465 22520 S  0% threaded­ml
22465 22521 S  0% PulseaudioSound
2338     – S  1% /usr/bin/gnome­shell
2338  2338 S  1% gnome­shell

PID    → process ID
TID    → thread ID
S       → status (R=running/runnable, S=interruptible sleep, D=uninterruptible sleep,
Z=zombie, T=stopped or being traced, W=swapping, E=exited)
CPU   → percentage of CPU time used
CMD  → command or binary being executed

 

We can get the user pane by pressing the u key:

NPROCS SYSCPU USRCPU  VSIZE   RSIZE  PSIZE SWAPSZ RDDSK  WRDSK RNET SNET CPU RUID
82  0.18s  1.37s  50.3G    4.4G   3.2G     8K    0K 310.8M   35   38 25% marc
279  0.22s  0.07s  10.5G  689.5M 433.2M     0K    0K  4192K    0    0  5% root
1  0.00s  0.00s 620.6M  18264K 11512K     0K    0K     0K    0    0  0% polkitd
1  0.00s  0.00s 125.2M  11296K  8141K     0K    0K     0K    0    0  0% dbus
1  0.00s  0.00s 392.3M  10320K  3050K     0K    0K     0K    0    0  0% colord
1  0.00s  0.00s 113.1M   3948K   647K     0K    0K     0K    0    0  0% chrony
2  0.00s  0.00s 56040K   3108K   816K     0K    0K     0K    0    0  0% avahi
1  0.00s  0.00s 20752K   2740K   717K     0K    0K     0K    0    0  0% nobody
1  0.00s  0.00s 160.8M   2392K   291K     0K    0K     0K    0    0  0% rtkit
1  0.00s  0.00s 58172K   2304K   756K     0K    0K     0K    0    0  0% rpc

NPROCS   → number of child processes forked
SYSCPU   → system cpu time used
USRCPU   → user cpu time used
VSIZE       → virtual memory used
RSIZE       → resident memory used
PSIZE       → approximate unshared resident memory
SWAPSZ  → swap space used
RDDSK    → physical I/O of this process and all its children
WRDSK    → physical I/O of this process and all its children
RNET        → number of TCP + UDP packets received (kernel module netatop is required)
SNET        → number of TCP + UDP packets transmitted (kernel module netatop is required)
CPU          → percentage of CPU used by this process
CMD         → command or binary being executed

 

We can get the miscellaneous pane by pressing the v key:

 PID  TID PPID ENVID RUID RGID EUID EGID SUID SGID FSUID FSGID STDATE     STTIME   ENDATE ENTIME ST EXC S CPU CMD
8435    –   ­ 1     0 marc marc marc marc marc marc marc   marc 2015/10/22 22:06:58 active active — – ­­ ­ S 27% insy
8435 8435    1     0 marc marc marc marc marc marc marc   marc 2015/10/22 22:06:58 active active — – ­­ ­ S  2% insy
8435 8436    1     0 marc marc marc marc marc marc marc   marc 2015/10/22 22:06:58 active active — – ­­ ­ D 25% insy
8435 8450    1     0 marc marc marc marc marc marc marc   marc 2015/10/22 22:06:59 active active — – ­­ ­ S  0% dco
8435 8451    1     0 marc marc marc marc marc marc marc   marc 2015/10/22 22:06:59 active active — – ­­ ­ S  0% gdb
8435 8455    1     0 marc marc marc marc marc marc marc   marc 2015/10/22 22:06:59 active active — – ­­ ­ S  0% ins
8435 8456    1     0 marc marc marc marc marc marc marc   marc 2015/10/22 22:06:59 active active — – ­­ ­ S  0% ins
8435 8463    1     0 marc marc marc marc marc marc marc   marc 2015/10/22 22:06:59 active active — – ­­ ­ S  0% QTh

PID         → process ID
TID         → thread ID
PPID       → parent PID
ENVID    → not used
RUID      → real UID
RGID      → real GID
EUID      → effective user ID
EGID      → effective GID
SUID      → saved UID
SGID      → saved GID
FSUID    → filesystem user­id under which this process executes
FSGID    → filesystem group­id under which this process executes
STDATE  → start date
STTIME  → start time
ENDATE → end date
ENTIME  → end time
ST           → status of the process (N=new process, E=exit voluntarily, S=signal terminated,
C=signal terminated + core dump produced)
EXC        → exit code (only for terminated processes)
S             → status (R=running/runnable, S=interruptible sleep, D=uninterruptible sleep,
Z=zombie, T=stopped or being traced, W=swapping, E=exited)
CPU       → percentage of CPU used by this process
CMD       → command or binary being executed

 

To show our own customised layout we should press the o key. We shall see a bit further down how to do that.

We can sort the data in the lower pane by pressing C, M, D, N or A:

C → sort by CPU consumption (default)

M → sort by resident memory consumption

D → sort by disk I/O

N → sort by network I/O

A → sort by resource consumption (ponderation of cpu, memory, disk & network)

We can also perform a few extra actions with the keys …

R → calculate proportionate process size (PSIZE)

x → toggle colours on/off

z → freeze the statistics on screen to allow detailed examination

i → change interval from default 10 seconds

t → refresh the statistics in the lower pane

r → reset all counters to 0 to see activity since last boot

U → filter out processes not owned by the username regex (remove filter with U+Return)

P → filter out processes whose name does not match the given regex

S → filter out all LVM/disk/net not matching the given regex

a → toggle “all users” or “active users”

G → toggles “all processes” or “alive processes”

f → toggles “all resources” or “active resources” in the top pane

F → switches on/off sorting of resources by utilisation in top pane

1 → switches the top pane between per interval and per second

l → limits the lines shown in top pane

k → kills the highlighted process

q → quits atop

PgUp, PgDn, up/down arrows → scroll the lower pane up and down

We can create our own customised layout by editing the line ownprocline in the configuration file /etc/atoprc or ~/.atoprc. Let’s see how to do that by looking at an example:

root:~> cat ~/.atoprc
flags           CafF1
interval        5
#username
#procname
maxlinecpu      0
maxlinelvm      3
maxlinemdd      1
maxlinedisk     2
maxlineintf     3
cpucritperc    80
dskcritperc    80
netcritperc    80
memcritperc    80
swpcritperc    80
swoutcritsec   10
almostcrit     90
ownprocline    PID:50 VGROW:40 RGROW:45 COMMAND­LINE:50
ownsysprcline  PRCSYS:8 PRCUSER:8 BLANKBOX:0 PRCNPROC:7 PRCNZOMBIE:5 PRCCLONES:4
BLANKBOX:0 PRCNNEXIT:6
ownallcpuline  CPUSYS:8 CPUUSER:7 CPUIRQ:4 BLANKBOX:0
CPUIDLE:5 CPUWAIT:6 BLANKBOX:0 CPUSTEAL:1 CPUGUEST:3
ownonecpuline  CPUISYS:8 CPUIUSER:7 CPUIIRQ:4 BLANKBOX:0
CPUIIDLE:5 CPUIWAIT:6 BLANKBOX:0 CPUISTEAL:1 CPUIGUEST:3
owncplline     CPLAVG1:4 CPLAVG5:3 CPLAVG15:2 BLANKBOX:0 CPLCSW:6
CPLINTR:5 BLANKBOX:0 CPLNUMCPU:1
ownmemline     MEMTOT:2 MEMFREE:5 MEMCACHE:3 MEMDIRTY:1
MEMBUFFER:3 MEMSLAB:3 BLANKBOX:0 BLANKBOX:0 BLANKBOX:0
BLANKBOX:0
ownswpline     SWPTOT:3 SWPFREE:4 BLANKBOX:0 BLANKBOX:0
BLANKBOX:0 BLANKBOX:0 BLANKBOX:0 BLANKBOX:0 SWPCOMMITTED:5
SWPCOMMITLIM:6
ownpagline     PAGSCAN:3 PAGSTALL:1 BLANKBOX:0 PAGSWIN:4
PAGSWOUT:3
owndskline     DSKNAME:8 DSKBUSY:7 DSKNREAD:6 DSKNWRITE:6
DSKKBPERRD:4 DSKKBPERWR:4 DSKMBPERSECRD:5 DSKMBPERSECWR:5
DSKAVQUEUE:1 DSKAVIO:5
ownnettrline   NETTRANSPORT:9 NETTCPI:8 NETTCPO:8 NETUDPI:8
NETUDPO:8 NETTCPACTOPEN:6 NETTCPPASVOPEN:5 NETTCPRETRANS:4
NETTCPINERR:3 NETTCPORESET:20 NETUDPNOPORT:1 NETUDPINERR:3
ownnetnetline  NETNETWORK:5 NETIPI:4 NETIPO:4 NETIPFRW:4
NETIPDELIV:4 BLANKBOX:0 BLANKBOX:0 BLANKBOX:0 NETICMPIN:1
NETICMPOUT:1
ownnetifline   NETNAME:8 NETPCKI:7 NETPCKO:7 NETSPEEDIN:6
NETSPEEDOUT:6 NETCOLLIS:3 NETMULTICASTIN:2 NETRCVERR:5
NETSNDERR:5 NETRCVDROP:4 NETSNDDROP:4

1. flags → we can use “g, m, d, n, u, p, s, c & v” to specify the lower pane we want
   we can use “A, C, D, M & N” to specify the sorting criteria
   we can use “a, f, 1 & x” to toggle on/off various options explained above

   interval → default is 10 seconds but we can change it to any reasonable value

   username → regex with the username whose processes we want shown by default

   procname → regex with the process name whose details we want shown by default

   maxline* → maximum number of lines we want shown in the top pane for the various resources

   *critperc → utilisation threshold above which the line flashes red

   swoutcritsec → pages swapped out per second above which the line flashes red

   almostcrit → percentage of critical threshold above which the line flashes blue

   own*line → customise the format of columns by specifying the fields you want and their priority
   in case they do not fit in the screen

To gather the network statistics on a per process basis we need to install an extra kernel module called netatop. This kernel module can be downloaded from http://www.atoptool.nl. Additionally, we need to install 2 packages that are pre-requisites for netatop:

# dnf -y install kernel­-devel zlib­-devel

Once we have installed the 2 extra packages and downloaded netatop, we have to perform the usual drill:

gunzip
tar xvf
cd netatop
make
make install
modprobe netatop

As soon as the kernel module is available, we should be able to use the per process network options.

 

 

<< htop                     prtstat >>