atop is in functionality and aim very similar to top and htop. It can be used interactively and in the background dumping gathered stats into binary files. However, as there are other tools (e.g. sar) that do the data gathering & dumping really well, we shall focus only in the interactive aspect of atop which is the one most people use.
As it the case with htop, atop needs to be manually installed as it does not come by default:
# dnf -y install atop
To launch the interactive ncurses GUI we just execute atop in the command line:
In the top frame, from left-to-right we can see:
PRC | sys 0.38s | user 2.26s
Number of seconds spent on sys and user mode by all the cores in the last time interval. If the interval is kept to the default 10 seconds, 0.38 + 2.26 seconds the cores were busy and 7.36 they were waiting for I/O to complete or idle.
#proc 369 | #trun 2 | #tslpi 700 | #tslpu 0 | #zombie 2 | clones 19 | | #exit 16
Number of current processes, threads running or runnable, threads in interruptible sleep,
threads in uninterruptible sleep, zombie processes and processes that exited in the last
interval.
CPU | sys 5% | user 26% | irq 0% | | idle 74% | wait 24% | | steal 0% | guest 0% | curf 3.25GHz | curscal 95%
Percentage of all CPU time spent on system, user, interrupt, idle, I/O wait, steal & guest modes.
The steal mode applies only to VMs as it implies the time stolen by other VMs running on the same system.
The guest applies only to the VM host and it shows the time used by its VM guests.
The curf and curscal values shows the average frequency and scaling of all the cores. If the CPU architecture allows it, the frequency of cores is throttled down by many OSes where the load is low in order to save power. The CPU utilisation figures already take this into account. For example: 50% utilisation of a core at 50% max frequency = 25% effective utilisation.
cpu | sys 2% | user 13% | irq 0% | | idle 67% | cpu004 w 17% | | steal 0% | guest 0% | curf 3.27GHz | curscal 96%
All the lines starting with “cpu” in lowercase show the stats for a particular core.
CPL | avg1 1.41 | avg5 1.24 | | avg15 1.22 | | csw 16157 | | intr 12268 | | | numcpu 8
Average load in the last 1, 5 and 15 minutes, number of context switches, number of interrupts and number of cores available.
MEM | tot 15.5G | free 3.3G | cache 8.6G | dirty 5.1M | buff 137.8M | slab 408.1M | | slrec 288.3M | shmem 460.9M | shrss 21.3M | vmbal 0.0M | hptot 0.0M | hpuse 0.0M
Total amount of memory available, free memory, cached memory, dirty memory that is due to be written to disk, I/O buffers size, kernel memory size, kernel reclaimable memory, shared memory, resident shared memory, swapped memory, Vmware’s baloon driver memory, hugepages total memory and hugepages used memory.
SWP | tot 3.8G | free 3.8G | | | | | | | | | | | vmcom 6.6G | vmlim 11.5G | |
Total swap space available, free swap space, virtual memory commited space and virtual memory max limit (default vlim = swap available + 50% of RAM)
PAG | scan 0 | stall 0 | swin 0 | swout 0
This line is not shown in the screenshot above because no paging occurred. But the figures are for number of pages scanned and number of stalls. The kernel scans memory looking for pages it can reclaim when the free memory falls below a certain threshold (stall event).
Swin and swout are the number of pages pulled from swap into memory (former) and number of pages pushed from memory onto swap (latter).
LVM/MDD/DSK | sdb | busy 71% || read 0 | write 1381 || KiB/r 0 | KiB/w 15 ||
MBr/s 0.00 | MBw/s 2.04 || avq 3.59 | avio 5.11 ms ||
Name of logical volume, software RAID volume or disk, percentage of time the device was busy handling I/O, read & write requests, Kbytes and Mbytes read/written, average I/O queue size and average service time per I/O request (seek + latency + data transfer).
NET | transport | tcpi 4987 | tcpo 5018 | | udpi 1301 | udpo 1164 | tcpao 362 | tcppo 0 | | tcprs 26 | tcpie 11 | tcpor 22 | udpnp 3 | | udpie 0 |
Number of TCP packets received and transmitted, UDP packets received and transmitted, TCP connections active and passive, TCP retransmissions, TCP input and output errors, UDP no ports and input errors. Figures shown are IPv4 + IPv6.
NET | network | ipi 6327 | | ipo 6211 | ipfrw 0 | | deliv 6320 | | | | | | icmpi 0 | icmpo 0
Number of IP packets received, forwarded and sent, and number of ICMP packets received and sent.
NET | wlo1 0% | pcki 6178 | | pcko 6114 | si 20 Kbps | so 5 Kbps | | coll 0 | mlti 0 | | erri 0 | erro 0 | drpi 0 | | drpo 0 |
Interface name, input & output packets, bits received & sent, collisions, received multicast packets, input & output errors and dropped input & output packets.
The lower pane shows information related to processes & threads. By default the generic pane is shown, but if it isn’t and we want to go back to it we can do so pressing the g key:
PID TID RUID EUID THR SYSCPU USRCPU VGROW RGROW RDDSK WRDSK ST EXC S CPUNR CPU CMD
8435 – marc marc 9 0.13s 1.02s 0K 20K 0K 9104K — – S 3 23% insync
2338 – marc marc 7 0.02s 0.08s 0K 0K 0K 0K – – S 1 2% gnome
1535 – root root 4 0.04s 0.04s 0K 0K 0K 0K – – S 0 2% Xorg.bin
2681 – marc marc 4 0.00s 0.04s 0K 0K 0K 0K — – S 4 1% gnome
-
PID -> process ID
TID -> thread ID
RUID -> real user ID
EUID -> effective user ID
THR -> number of threads
SYSCPU -> system cpu time used
USRCPU -> user cpu time used
VGROW -> virtual memory growth in last interval
RGROW -> resident memory growth in the last interval
RDDSK -> physical read I/O of this process and all its children
WRDSK -> physical written I/O of this process and all its children
ST -> status of the process (N=new process, E=exit voluntarily, S=signal terminated,
C=signal terminated + core dump produced)
EXC -> exit code of process (usually 0 → no problem, >0 → problem somewhere)
S -> status (R=running/runnable, S=interruptible sleep, D=uninterruptible sleep,
Z=zombie, T=stopped or being traced, W=swapping, E=exited)
CPUNR -> cpu ID process is or has been running on
CPU -> percentage of cpu used
CMD -> command or binary being executed
We can get the process pane by pressing the p key:
NPROCS SYSCPU USRCPU VSIZE RSIZE PSIZE SWAPSZ RDDSK WRDSK RNET SNET CPU CMD
1 50.47s 6m05s 2.7G 209.7M 0K 0K 8K 3.3G 0 0 25% insync
1 1.44s 15.41s 2.2G 751.7M 0K 0K 0K 120K 0 0 1% gnomeshell
1 0.78s 15.02s 1.7G 360.0M 0K 0K 19572K 96532K 0 0 1% soffice.bin
8 2.45s 12.43s 7.6G 1.2G 0K 0K 29480K 227.5M 0 0 1% chrome
1 6.61s 7.01s 537.6M 92184K 0K 0K 0K 0K 0 0 1% Xorg.bin
Most stats in this layout are cumulative since process startup and not per interval!
NPROCS → number of child processes forked
SYSCPU → system cpu time used
USRCPU → user cpu time used
VSIZE → virtual memory used
RSIZE → resident memory used
PSIZE → approximate unshared resident memory
SWAPSZ → swap space used
RDDSK → physical I/O of this process and all its children
WRDSK → physical I/O of this process and all its children
RNET → number of TCP + UDP packets received (kernel module netatop is required)
SNET → number of TCP + UDP packets transmitted (kernel module netatop is required)
CPU → percentage of CPU used by this process
CMD → command or binary being executed
We can get the memory pane by pressing the m key:
PID TID MINFLT MAJFLT VSTEXT VSLIBS VDATA VSTACK VSIZE RSIZE PSIZE VGROW RGROW SWAPSZ RUID EUID MEM CMD
2338 – 0 0 16K 68424K 1.5G 184K 2.2G 752.2M 0K 0K 0K 0K marc marc 5% gnomeshell
5001 – 305 0 90844K 37960K 858.3M 136K 1.5G 518.2M 0K 0K 72K 0K marc marc 3% chrome
11022 – 14 0 90844K 50584K 701.6M 136K 1.2G 241.2M 0K 0K 0K 0K marc marc 2% chrome
22465 – 0 0 31268K 104.0M 471.2M 136K 1.2G 213.0M 0K 0K 0K 0K marc marc 1% spotify
PID → process ID
TID → thread ID
MINFLT → number of minor page faults
MAJFLT → number of major page faults
VSTEXT → virtual shared memory used for text segments
VSLIBS → virtual shared memory used for library segments
VDATA → virtual memory used for data
VSTACK → virtual memory used for stack
VSIZE → total virtual memory size
RSIZE → resident memory size
PSIZE → process memory size (you need to press R to get this figure!)
VGROW → virtual memory growth in last interval
RGROW → resident memory growth in last interval
SWAPSZ → swap memory used
RUID → real user ID
EUID → effective user ID
MEM → percentage of total memory used by this process
CMD → command or binary being executed
We can get the device pane by pressing the d key:
PID TID RDDSK WRDSK WCANCL DSK CMD
8435 – 0K 10436K 0K 91% insync
2527 – 0K 504K 12K 4% trackerstore
2515 – 460K 0K 0K 4% trackerextrac
5001 – 0K 84K 4K 1% chrome
PID → process ID
TID → thread ID
RDDSK → physical I/O of this process and all its children
WRDSK → physical I/O of this process and all its children
WCANCL → physical I/O written and withdrawn
DSK → percentage of disk bandwidth used
CMD → command or binary being executed
We can get the network pane by pressing the n key if the netatop kernel module is installed:
PID TID TCPRCV TCPRASZ TCPSND TCPSASZ UDPRCV UDPRASZ UDPSND UDPSASZ BANDWI BANDWO NET CMD
5001 – 48 258 65 264 13 169 15 201 23 Kbps 32 Kbps 100% chrome
8435 - 0 0 0 0 0 0 0 0 0 Kbps 0 Kbps 0% insync
1535 – 0 0 0 0 0 0 0 0 0 Kbps 0 Kbps 0% Xorg.bin
5148 – 0 0 0 0 0 0 0 0 0 Kbps 0 Kbps 0% chrome
2338 – 0 0 0 0 0 0 0 0 0 Kbps 0 Kbps 0% gnomeshe
PID → process ID
TID → thread ID
ENVID → field not used
TRUN → threads running or runnable
TSLPI → threads in interruptible sleep
TSLPU → threads in uninterruptible sleep
POLI → scheduling policy: normal, batch, idle, fifo and roundrobin
NICE → niceness level from 20 (highest) to +19 (lowest)
PRI → priority level from 0 (highest) to 139 (lowest)
RPTR → realtime priority: 0 for timesharing classes (normal/batch/idle) and 1 (lowest) to 99
(highest) for process in realtime class (roundrobin/fifo)
CPUNR → cpu ID process is or has been running on
ST → status of the process (N=new process, E=exit voluntarily, S=signal terminated,
C=signal terminated + core dump produced)
EXC → exit code (only for terminated processes)
S → status (R=running/runnable, S=interruptible sleep, D=uninterruptible sleep,
Z=zombie, T=stopped or being traced, W=swapping, E=exited)
CPU → percentage of CPU time used
CMD → command or binary being executed
We can get the wide command line pane by pressing the c key:
PID TID S CPU COMMAND-LINE (horizontal scroll with < and > keys)
8435 – S 28% ./insync start
7913 – R 4% atop
22465 – S 1% /usr/lib64/spotifyclient/spotify
2338 - S 1% /usr/bin/gnomeshell
2681 – S 0% /usr/libexec/gnometerminalserver
2285 – S 0% /usr/bin/pulseaudio start
786 – S 0% irq/32iwlwifi
11022 – S 0% /opt/google/chrome/chrome type=renderer lang=enUS –forcefieldtrials=*AffiliationBasedMatchi
1535 – S 0% /usr/libexec/Xorg.bin :0 background none noreset verbose 3 logfile /dev/null auth
PID → process ID
TID → thread ID
S → status (R=running/runnable, S=interruptible sleep, D=uninterruptible sleep,
Z=zombie, T=stopped or being traced, W=swapping, E=exited)
CPU → percentage of CPU time used
CMD → command or binary being executed
We can get the thread pane by pressing the y key:
PID TID S CPU COMMAND-LINE (horizontal scroll with < and > keys)
8435 – S 28% ./insync start
8435 8435 S 2% insync
8435 8436 D 25% insync
8435 8450 S 0% dconf worker
8435 8451 S 0% gdbus
8435 8455 S 0% insync
8435 8456 S 0% insync
8435 8463 S 0% QThread
8435 8465 S 0% insync
8435 8466 S 0% QThread
7913 - R 4% atop
22465 - S 1% /usr/lib64/spotifyclient/spotify
22465 22497 S 0% Network Thread
22465 22520 S 0% threadedml
22465 22521 S 0% PulseaudioSound
2338 – S 1% /usr/bin/gnomeshell
2338 2338 S 1% gnomeshell
PID → process ID
TID → thread ID
S → status (R=running/runnable, S=interruptible sleep, D=uninterruptible sleep,
Z=zombie, T=stopped or being traced, W=swapping, E=exited)
CPU → percentage of CPU time used
CMD → command or binary being executed
We can get the user pane by pressing the u key:
NPROCS SYSCPU USRCPU VSIZE RSIZE PSIZE SWAPSZ RDDSK WRDSK RNET SNET CPU RUID
82 0.18s 1.37s 50.3G 4.4G 3.2G 8K 0K 310.8M 35 38 25% marc
279 0.22s 0.07s 10.5G 689.5M 433.2M 0K 0K 4192K 0 0 5% root
1 0.00s 0.00s 620.6M 18264K 11512K 0K 0K 0K 0 0 0% polkitd
1 0.00s 0.00s 125.2M 11296K 8141K 0K 0K 0K 0 0 0% dbus
1 0.00s 0.00s 392.3M 10320K 3050K 0K 0K 0K 0 0 0% colord
1 0.00s 0.00s 113.1M 3948K 647K 0K 0K 0K 0 0 0% chrony
2 0.00s 0.00s 56040K 3108K 816K 0K 0K 0K 0 0 0% avahi
1 0.00s 0.00s 20752K 2740K 717K 0K 0K 0K 0 0 0% nobody
1 0.00s 0.00s 160.8M 2392K 291K 0K 0K 0K 0 0 0% rtkit
1 0.00s 0.00s 58172K 2304K 756K 0K 0K 0K 0 0 0% rpc
NPROCS → number of child processes forked
SYSCPU → system cpu time used
USRCPU → user cpu time used
VSIZE → virtual memory used
RSIZE → resident memory used
PSIZE → approximate unshared resident memory
SWAPSZ → swap space used
RDDSK → physical I/O of this process and all its children
WRDSK → physical I/O of this process and all its children
RNET → number of TCP + UDP packets received (kernel module netatop is required)
SNET → number of TCP + UDP packets transmitted (kernel module netatop is required)
CPU → percentage of CPU used by this process
CMD → command or binary being executed
We can get the miscellaneous pane by pressing the v key:
PID TID PPID ENVID RUID RGID EUID EGID SUID SGID FSUID FSGID STDATE STTIME ENDATE ENTIME ST EXC S CPU CMD
8435 – 1 0 marc marc marc marc marc marc marc marc 2015/10/22 22:06:58 active active — – S 27% insy
8435 8435 1 0 marc marc marc marc marc marc marc marc 2015/10/22 22:06:58 active active — – S 2% insy
8435 8436 1 0 marc marc marc marc marc marc marc marc 2015/10/22 22:06:58 active active — – D 25% insy
8435 8450 1 0 marc marc marc marc marc marc marc marc 2015/10/22 22:06:59 active active — – S 0% dco
8435 8451 1 0 marc marc marc marc marc marc marc marc 2015/10/22 22:06:59 active active — – S 0% gdb
8435 8455 1 0 marc marc marc marc marc marc marc marc 2015/10/22 22:06:59 active active — – S 0% ins
8435 8456 1 0 marc marc marc marc marc marc marc marc 2015/10/22 22:06:59 active active — – S 0% ins
8435 8463 1 0 marc marc marc marc marc marc marc marc 2015/10/22 22:06:59 active active — – S 0% QTh
PID → process ID
TID → thread ID
PPID → parent PID
ENVID → not used
RUID → real UID
RGID → real GID
EUID → effective user ID
EGID → effective GID
SUID → saved UID
SGID → saved GID
FSUID → filesystem userid under which this process executes
FSGID → filesystem groupid under which this process executes
STDATE → start date
STTIME → start time
ENDATE → end date
ENTIME → end time
ST → status of the process (N=new process, E=exit voluntarily, S=signal terminated,
C=signal terminated + core dump produced)
EXC → exit code (only for terminated processes)
S → status (R=running/runnable, S=interruptible sleep, D=uninterruptible sleep,
Z=zombie, T=stopped or being traced, W=swapping, E=exited)
CPU → percentage of CPU used by this process
CMD → command or binary being executed
To show our own customised layout we should press the o key. We shall see a bit further down how to do that.
We can sort the data in the lower pane by pressing C, M, D, N or A:
C → sort by CPU consumption (default)
M → sort by resident memory consumption
D → sort by disk I/O
N → sort by network I/O
A → sort by resource consumption (ponderation of cpu, memory, disk & network)
We can also perform a few extra actions with the keys …
R → calculate proportionate process size (PSIZE)
x → toggle colours on/off
z → freeze the statistics on screen to allow detailed examination
i → change interval from default 10 seconds
t → refresh the statistics in the lower pane
r → reset all counters to 0 to see activity since last boot
U → filter out processes not owned by the username regex (remove filter with U+Return)
P → filter out processes whose name does not match the given regex
S → filter out all LVM/disk/net not matching the given regex
a → toggle “all users” or “active users”
G → toggles “all processes” or “alive processes”
f → toggles “all resources” or “active resources” in the top pane
F → switches on/off sorting of resources by utilisation in top pane
1 → switches the top pane between per interval and per second
l → limits the lines shown in top pane
k → kills the highlighted process
q → quits atop
PgUp, PgDn, up/down arrows → scroll the lower pane up and down
We can create our own customised layout by editing the line ownprocline in the configuration file /etc/atoprc or ~/.atoprc. Let’s see how to do that by looking at an example:
root:~> cat ~/.atoprc
flags CafF1
interval 5
#username
#procname
maxlinecpu 0
maxlinelvm 3
maxlinemdd 1
maxlinedisk 2
maxlineintf 3
cpucritperc 80
dskcritperc 80
netcritperc 80
memcritperc 80
swpcritperc 80
swoutcritsec 10
almostcrit 90
ownprocline PID:50 VGROW:40 RGROW:45 COMMANDLINE:50
ownsysprcline PRCSYS:8 PRCUSER:8 BLANKBOX:0 PRCNPROC:7 PRCNZOMBIE:5 PRCCLONES:4
BLANKBOX:0 PRCNNEXIT:6
ownallcpuline CPUSYS:8 CPUUSER:7 CPUIRQ:4 BLANKBOX:0
CPUIDLE:5 CPUWAIT:6 BLANKBOX:0 CPUSTEAL:1 CPUGUEST:3
ownonecpuline CPUISYS:8 CPUIUSER:7 CPUIIRQ:4 BLANKBOX:0
CPUIIDLE:5 CPUIWAIT:6 BLANKBOX:0 CPUISTEAL:1 CPUIGUEST:3
owncplline CPLAVG1:4 CPLAVG5:3 CPLAVG15:2 BLANKBOX:0 CPLCSW:6
CPLINTR:5 BLANKBOX:0 CPLNUMCPU:1
ownmemline MEMTOT:2 MEMFREE:5 MEMCACHE:3 MEMDIRTY:1
MEMBUFFER:3 MEMSLAB:3 BLANKBOX:0 BLANKBOX:0 BLANKBOX:0
BLANKBOX:0
ownswpline SWPTOT:3 SWPFREE:4 BLANKBOX:0 BLANKBOX:0
BLANKBOX:0 BLANKBOX:0 BLANKBOX:0 BLANKBOX:0 SWPCOMMITTED:5
SWPCOMMITLIM:6
ownpagline PAGSCAN:3 PAGSTALL:1 BLANKBOX:0 PAGSWIN:4
PAGSWOUT:3
owndskline DSKNAME:8 DSKBUSY:7 DSKNREAD:6 DSKNWRITE:6
DSKKBPERRD:4 DSKKBPERWR:4 DSKMBPERSECRD:5 DSKMBPERSECWR:5
DSKAVQUEUE:1 DSKAVIO:5
ownnettrline NETTRANSPORT:9 NETTCPI:8 NETTCPO:8 NETUDPI:8
NETUDPO:8 NETTCPACTOPEN:6 NETTCPPASVOPEN:5 NETTCPRETRANS:4
NETTCPINERR:3 NETTCPORESET:20 NETUDPNOPORT:1 NETUDPINERR:3
ownnetnetline NETNETWORK:5 NETIPI:4 NETIPO:4 NETIPFRW:4
NETIPDELIV:4 BLANKBOX:0 BLANKBOX:0 BLANKBOX:0 NETICMPIN:1
NETICMPOUT:1
ownnetifline NETNAME:8 NETPCKI:7 NETPCKO:7 NETSPEEDIN:6
NETSPEEDOUT:6 NETCOLLIS:3 NETMULTICASTIN:2 NETRCVERR:5
NETSNDERR:5 NETRCVDROP:4 NETSNDDROP:4
-
flags → we can use “g, m, d, n, u, p, s, c & v” to specify the lower pane we want
we can use “A, C, D, M & N” to specify the sorting criteria
we can use “a, f, 1 & x” to toggle on/off various options explained aboveinterval → default is 10 seconds but we can change it to any reasonable value
username → regex with the username whose processes we want shown by default
procname → regex with the process name whose details we want shown by default
maxline* → maximum number of lines we want shown in the top pane for the various resources
*critperc → utilisation threshold above which the line flashes red
swoutcritsec → pages swapped out per second above which the line flashes red
almostcrit → percentage of critical threshold above which the line flashes blue
own*line → customise the format of columns by specifying the fields you want and their priority
in case they do not fit in the screen
To gather the network statistics on a per process basis we need to install an extra kernel module called netatop. This kernel module can be downloaded from http://www.atoptool.nl. Additionally, we need to install 2 packages that are pre-requisites for netatop:
# dnf -y install kernel-devel zlib-devel
Once we have installed the 2 extra packages and downloaded netatop, we have to perform the usual drill:
gunzip
tar xvf
cd netatop
make
make install
modprobe netatop
As soon as the kernel module is available, we should be able to use the per process network options.