Using the pmap Command in Linux

Pmap provide memory map of a process, The pmap command display the memory usage map of a process or multiple processes. Pmap reports information about the address space or memory usage map of a process. Pmap is actually a Sun OS command and Linux supports only very limited number of features. But it is very helpful for finding the complete address space of a process. To check memory usage of process we need PID or unique process ID of running process, we can get PID from /proc or regular commands like top or ps.

The following example displays the memory map of the current bash shell. In this example, 5732 is the PID of the bash shell.

$ pmap 5732
5732:   -bash
00393000    104K r-x--  /lib/ld-2.5.so
003b1000   1272K r-x--  /lib/libc-2.5.so
00520000      8K r-x--  /lib/libdl-2.5.so
0053f000     12K r-x--  /lib/libtermcap.so.2.0.8
0084d000     76K r-x--  /lib/libnsl-2.5.so
00c57000     32K r-x--  /lib/libnss_nis-2.5.so
00c8d000     36K r-x--  /lib/libnss_files-2.5.so
b7d6c000   2048K r----  /usr/lib/locale/locale-archive
bfd10000     84K rw---    [ stack ]
 total     4796K

pmap -x gives some additional information about the memory maps.

$  pmap -x 5732
5732:   -bash
Address   Kbytes     RSS    Anon  Locked Mode   Mapping
00393000     104       -       -       - r-x--  ld-2.5.so
003b1000    1272       -       -       - r-x--  libc-2.5.so
00520000       8       -       -       - r-x--  libdl-2.5.so
0053f000      12       -       -       - r-x--  libtermcap.so.2.0.8
0084d000      76       -       -       - r-x--  libnsl-2.5.so
00c57000      32       -       -       - r-x--  libnss_nis-2.5.so
00c8d000      36       -       -       - r-x--  libnss_files-2.5.so
b7d6c000    2048       -       -       - r----  locale-archive
bfd10000      84       -       -       - rw---    [ stack ]
-------- ------- ------- ------- -------
total kB    4796       -       -       -

To display the device information of the process maps use ‘pamp -d pid’.

Syntax or usage

#pmap PID

or

#pmap [options] PID

In outout it display total address, kbytes, mode and mapping.

Options

-x extended Show the extended format.
-d device Show the device format.
-q quiet Do not display some header/footer lines.
-V show version Displays version of program.

Memory usage map of single process

[root@info ~]# pmap 1013

1013: /usr/sbin/sshd
00110000 1480K r-x– /usr/lib/libcrypto.so.1.0.0
00282000 80K rw— /usr/lib/libcrypto.so.1.0.0
00296000 12K rw— [ anon ]
00299000 36K r-x– /lib/libkrb5support.so.0.1
002a2000 4K rw— /lib/libkrb5support.so.0.1
002a3000 16K r-x– /lib/libplc4.so
002a7000 4K rw— /lib/libplc4.so
002ab000 88K r-x– /lib/libaudit.so.1.0.0
002c1000 4K r—- /lib/libaudit.so.1.0.0
002c2000 4K rw— /lib/libaudit.so.1.0.0
002c3000 216K r-x– /lib/libgssapi_krb5.so.2.2
002f9000 4K rw— /lib/libgssapi_krb5.so.2.2
002fa000 808K r-x– /lib/libkrb5.so.3.3
003c4000 24K rw— /lib/libkrb5.so.3.3
003ca000 152K r-x– /lib/libk5crypto.so.3.1
003f0000 4K rw— /lib/libk5crypto.so.3.1
003f1000 92K r-x– /usr/lib/libnssutil3.so
00738000 4K r—- /lib/libresolv-2.12.so
00739000 4K rw— /lib/libresolv-2.12.so
0073a000 8K rw— [ anon ]
00825000 120K r-x– /lib/ld-2.12.so
00843000 4K r—- /lib/ld-2.12.so
00844000 4K rw— /lib/ld-2.12.so
0090d000 32K r-x– /lib/libwrap.so.0.7.6
00915000 4K rw— /lib/libwrap.so.0.7.6
00948000 484K r-x– /usr/sbin/sshd
009c1000 8K rw— /usr/sbin/sshd
009c3000 20K rw— [ anon ]
009e0000 92K r-x– /lib/libpthread-2.12.so
009f7000 4K r—- /lib/libpthread-2.12.so

total 8232K

Memory usage map of multiple processes

We can check memory map of multiple processes by inserting multiple PIDs. Add multiple PIDs with adding space.

pmap 1013 1217 1118

Extended memory map about a process

[root@info ~]# pmap -x 1013
1013: /usr/sbin/sshd
Address Kbytes RSS Dirty Mode Mapping
00110000 1480 92 0 r-x– libcrypto.so.1.0.0
00282000 80 80 80 rw— libcrypto.so.1.0.0
00296000 12 8 4 rw— [ anon ]
00299000 36 0 0 r-x– libkrb5support.so.0.1
002a2000 4 4 4 rw— libkrb5support.so.0.1
002a3000 16 0 0 r-x– libplc4.so
002a7000 4 4 4 rw— libplc4.so
002ab000 88 4 0 r-x– libaudit.so.1.0.0
002c1000 4 4 4 r—- libaudit.so.1.0.0
002c2000 4 4 4 rw— libaudit.so.1.0.0
002c3000 216 4 0 r-x– libgssapi_krb5.so.2.2
002f9000 4 4 4 rw— libgssapi_krb5.so.2.2
002fa000 808 4 0 r-x– libkrb5.so.3.3
003c4000 24 24 24 rw— libkrb5.so.3.3
003ca000 152 4 0 r-x– libk5crypto.so.3.1
003f0000 4 4 4 rw— libk5crypto.so.3.1
003f1000 92 0 0 r-x– libnssutil3.so
00408000 12 12 12 rw— libnssutil3.so
0040b000 12 0 0 r-x– libplds4.so
0040e000 4 4 4 rw— libplds4.so

——– ——- ——- ——- ——-
total kB 8232 – – –

Here Address, Kbyte, Dirty, RSS, mode and mapping containd information as below

Extended and Device Format Fields

Address: start address of map
Kbytes: size of map in kilobytes
RSS: resident set size in kilobytes
Dirty: dirty pages (both shared and private) in kilobytes
Mode: permissions on map: read, write, execute, shared, private (copy on write)
Mapping: file backing the map, or ‘[ anon ]’ for allocated memory, or ‘[ stack ]’ for the program stack
Offset: offset into the file
Device: device name (major:minor)

Understanding The top Command in Linux

Top command displays all the running process in the system ordered by certain columns. This displays the information real-time.

You can kill a process without exiting from top. Once you’ve located a process that needs to be killed, press “k” which will ask for the process id, and signal to send. If you have the privilege to kill that particular PID, it will get killed successfully.

PID to kill: 1309
Kill PID 1309 with signal [15]:
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 1309 geek   23   0 2483m 1.7g  27m S    0 21.8  45:31.32 gagent
 1882 geek   25   0 2485m 1.7g  26m S    0 21.7  22:38.97 gagent
 5136 root    16   0 38040  14m 9836 S    0  0.2   0:00.39 nautilus

Use top -u to display a specific user processes only in the top command output.

$ top -u geek

While unix top command is running, press u which will ask for username as shown below.

Which user (blank for all): geek
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 1309 geek   23   0 2483m 1.7g  27m S    0 21.8  45:31.32 gagent
 1882 geek   25   0 2485m 1.7g  26m S    0 21.7  22:38.97 gagent

1. Show Processes Sorted by any Top Output Column – Press O

By default top command displays the processes in the order of CPU usage. When the top command is running, press M (upper-case) to display processes sorted by memory usage as shown below.

To sort top output by any column, Press O (upper-case O) , which will display all the possible columns that you can sort by as shown below.

Current Sort Field: P for window 1:Def
Select sort field via field letter, type any other key to return

a: PID = Process Id v: nDRT = Dirty Pages count
d: UID = User Id y: WCHAN = Sleeping in Function
e: USER = User Name z: Flags = Task Flags
……..
When the linux top command is running, Press R, which does the sort in reverse order.

Kill a Task Without Exiting From Top – Press k

Once you’ve located a process that needs to be killed, press ‘k’ which will ask for the process id, and signal to send.  If you have the privilege to kill that particular PID, it will get killed successfully.

 

 

PID to kill: 1309
Kill PID 1309 with signal [15]: 
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 1309 geek   23   0 2483m 1.7g  27m S    0 21.8  45:31.32 gagent
 1882 geek   25   0 2485m 1.7g  26m S    0 21.7  22:38.97 gagent
 5136 root    16   0 38040  14m 9836 S    0  0.2   0:00.39 nautilus

3. Renice a Unix Process Without Exiting From Top – Press r

Press r, if you want to just change the priority of the process (and not kill the process). This will ask PID for renice, enter the PID and priority.


PID to renice: 1309
Renice PID 1309 to value: 
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 1309 geek   23   0 2483m 1.7g  27m S    0 21.8  45:31.32 gagent
 1882 geek   25   0 2485m 1.7g  26m S    0 21.7  22:38.97 gagent

4. Display Selected User in Top Output Using top -u

Use top -u to display a specific user processes only in the top command output.

$ top -u geek

While unix top command is running, press u which will ask for username as shown below.

Which user (blank for all): geek
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 1309 geek   23   0 2483m 1.7g  27m S    0 21.8  45:31.32 gagent
 1882 geek   25   0 2485m 1.7g  26m S    0 21.7  22:38.97 gagent

Display Only Specific Process with Given PIDs Using top -p

Use top -p as shown below to display specific PIDs.

$ top -p 1309, 1882
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 1309 geek   23   0 2483m 1.7g  27m S    0 21.8  45:31.32 gagent
 1882 geek   25   0 2485m 1.7g  26m S    0 21.7  22:38.97 gagent

5. Display All CPUs / Cores in the Top Output – Press 1 (one)

Top output by default shows CPU line for all the CPUs combined together as shown below.

top - 20:10:39 up 40 days, 23:02,  1 user,  load average: 4.97, 2.01, 1.25
Tasks: 310 total,   1 running, 309 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.5%us,  0.7%sy,  0.0%ni, 92.3%id,  6.4%wa,  0.0%hi,  0.0%si,  0.0%st

Press 1 (one), when the top command is running, which will break the CPU down and show details for all the individual CPUs running on the system as shown below.

top - 20:10:07 up 40 days, 23:03,  1 user,  load average: 5.32, 2.38, 1.39
Tasks: 341 total,   3 running, 337 sleeping,   0 stopped,   1 zombie
Cpu0  :  7.7%us,  1.7%sy,  0.0%ni, 79.5%id, 11.1%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu1  :  0.3%us,  0.0%sy,  0.0%ni, 94.9%id,  4.7%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu2 :  3.3%us,  0.7%sy,  0.0%ni, 55.7%id, 40.3%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu3 :  5.0%us,  1.0%sy,  0.0%ni, 86.2%id,  7.4%wa,  0.0%hi,  0.3%si,  0.0%st
Cpu4  : 38.5%us,  5.4%sy,  0.3%ni,  0.0%id, 54.8%wa,  0.0%hi,  1.0%si,  0.0%st
Cpu5  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu6  :  0.3%us,  0.7%sy,  0.0%ni, 97.3%id,  1.7%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu7  :  5.4%us,  4.4%sy,  0.0%ni, 82.6%id,  7.7%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu8 :  1.7%us,  1.7%sy,  0.0%ni, 72.8%id, 23.8%wa,  0.0%hi,  0.0%si,  0.0%st

6. Refresh Unix Top Command Output On demand (or) Change Refresh Interval

By default, linux top command updates the output every 3.0 seconds. When you want to update the output on-demand, press space bar.

To change the output update frequency, press d in interactive mode, and enter the time in seconds as shown below.

Change delay from 3.0 to: 10
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 1309 geek   23   0 2483m 1.7g  27m S    0 21.8  45:31.32 gagent
 1882 geek   25   0 2485m 1.7g  26m S    0 21.7  22:38.97 gagent

Using the free command in Linux

Free command displays information about the physical (RAM) and swap memory of your system.

In the example below, the total physical memory on this system is 1GB. The values displayed below are in KB.

# free
       total   used    free   shared  buffers  cached
Mem: 1034624   1006696 27928  0       174136   615892
-/+ buffers/cache:     216668      817956
Swap:    2031608       0    2031608

The following example will display the total memory on your system including RAM and Swap.

In the following command:

  • option m displays the values in MB
  • option t displays the “Total” line, which is sum of physical and swap memory values
  • option o is to hide the buffers/cache line from the above example.
# free -mto
                  total       used      free     shared    buffers     cached
Mem:          1010        983         27              0         170           601
Swap:          1983            0    1983
Total:          2994        983     2011

Free command examples

Example 1: Display RAM details in Linux machine

free

Output:

total used free shared buffers cached
Mem: 8027952 4377300 3650652 0 103648 1630364
-/+ buffers/cache: 2643288 5384664
Swap: 15624188 608948 15015240

Let us see what this table for you.

Line 1: Indicates Memory details like total available RAM, used RAM, Shared RAM, RAM used for buffers, RAM used of caching content.
Line 2: Indicates total buffers/Cache used and free.
Line 3: Indicates total swap memory available, used swap and free swap memory size available.

Let us dig more in to these lines to better understand it as Linux user prospective.

Line 1:Mem: 8027952 4377300 3650652 0 103648 1630364

8027952 : Indicates memory/physical RAM available for your machine. These numbers are in KB’s
4377300 : Indicates memory/RAM used by system. This include even buffers and cached data size as well.
3650652 : Indicates Total RAM free and available for new process to run.
0 :  Indicates shared memory. This column is obsolete and may be removed in future releases of free.
103648 : Indicates total RAM buffered by different applications in Linux
1630364 : Indicates total RAM used for Caching of data for future purpose.

Puzzled with buffers and cache?

What is the difference between buffers and Cache?

A buffer is a temporary location to store data for a particular application and this data is not used by any other application. This is similar to bandwidth concept. When you try to send burst of data through network, if your network card is capable of sending less data, it will keep these huge amounts of data in buffer so that it can send data constantly in lesser speeds. In other hand Cache is a memory location to store frequently used data for faster access. Other difference between a buffer and a cache is that cache can be used multiple times where as buffer is used single time. And both are temporary store for your data processing.

Line 2: -/+ buffers/cache: 2643288 5384664

2643288 : This is actual size of used RAM which we get from RAM used -(buffers + cache)
A bit of mathematical calculation
Used RAM = +4377300
Used Buffers = -103648
Used Cache = -1630364
Actual Total used RAM is 4377300 -(103648+1630364)= 2643288

Then why my Linux machine is showing 4377300 as used RAM. This is because Linux counts cached RAM, Buffered RAM to this used RAM. But in future if any application want to use these buffers/cache, Linux will free it for you. To know more about this,visit this site.

5384664 : Indicates actual total RAM available, we get to this number by subtracting actual RAM used from total RAM available in the system.

Total RAM = +8027952
actual used RAM = -2643288
Total actual available RAM = 5384664

So from today on words don’t complaint that Linux ate your RAM, its our understanding of free command output which is the culprit and the teacher who thought us Linux. If any one asks what is the free RAM available, we have to give this number(5384664) instead of first line number(4377300) for free RAM available in your machine.

Line 3: Swap: 15624188 608948 15015240

This line indicates swap details like total SWAP size, used as well as free SWAP.

Swap is a virtual memory created on HDD to increase RAM size virtually.

Using the vmstat Command

vmstat reports information about processes, memory, paging, block IO, traps, disks andcpu activity.

The first report produced gives averages since the last reboot. Additional reports give information on a sampling period of length delay. The process and memory reports are instantaneous in either case.

$ vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0 305416 260688  29160 2356920    2    2     4     1    0    0  6  1 92  2  0

To execute vmstat every 2 seconds for 10 times, do the following. After executing 10 times, it will stop automatically.
$ vmstat 2 10
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0      0 537144 182736 6789320    0    0     0     0    1    1  0  0 100  0  0
 0  0      0 537004 182736 6789320    0    0     0     0   50   32  0  0 100  0  0
..

iostat and vmstat are part of the sar utility. You should install sysstat package to get iostat and vmstat working.

This article provides a total of 24 examples on iostat, vmstat, and mpstat commands.

  • iostat reports CPU, disk I/O, and NFS statistics.
  • vmstat reports virtual memory statistics.
  • mpstat reports processors statictics.

This article is part of our ongoing Linux performance monitoring series.

Please note that iostat and vmstat are part of the sar utility. You should install sysstat package as explained in our sar (sysstat) article to get iostat and vmstat working.

IOSTAT EXAMPLES

1. iostat – Basic example

Iostat without any argument displays information about the CPU usage, and I/O statistics about all the partitions on the system as shown below.

$ iostat
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db)       07/09/2011

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.68    0.00    0.52    2.03    0.00   91.76

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             194.72      1096.66      1598.70 2719068704 3963827344
sda1            178.20       773.45      1329.09 1917686794 3295354888
sda2             16.51       323.19       269.61  801326686  668472456
sdb             371.31       945.97      1073.33 2345452365 2661206408
sdb1            371.31       945.95      1073.33 2345396901 2661206408
sdc             408.03       207.05       972.42  513364213 2411023092
sdc1            408.03       207.03       972.42  513308749 2411023092

2. iostat – Display only cpu statistics

iostat option -c, displays only the CPU usage statistics as shown below.

$ iostat -c
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db)       07/09/2011

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.68    0.00    0.52    2.03    0.00   91.76

3. iostat – Display only disk I/O statistics

iostat option -d, displays only the disk I/O statistics as shown below.

$ iostat -d
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db)       07/09/2011

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             194.71      1096.61      1598.63 2719068720 3963827704
sda1            178.20       773.41      1329.03 1917686810 3295355248
sda2             16.51       323.18       269.60  801326686  668472456
sdb             371.29       945.93      1073.28 2345452365 2661209192
sdb1            371.29       945.91      1073.28 2345396901 2661209192
sdc             408.01       207.04       972.38  513364213 2411024484
sdc1            408.01       207.02       972.38  513308749 2411024484

4. iostat – Display only network statistics

iostat option -n, displays only the device and NFS statistics as shown below.

$ iostat -n
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db)        07/09/2011

avg-cpu:  %user   %nice    %sys %iowait   %idle
           4.33    0.01    1.16    0.31   94.19

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               2.83         0.35         5.39   29817402  457360056
sda1              3.32        50.18         4.57 4259963994  387641400
sda2              0.20         0.76         0.82   64685128   69718576
sdb               6.59        15.53        42.98 1318931178 3649084113
sdb1             11.80        15.53        42.98 1318713382 3649012985

Device:                  rBlk_nor/s   wBlk_nor/s   rBlk_dir/s   wBlk_dir/s   rBlk_svr/s   wBlk_svr/s
192.168.1.4:/home/data      90.67        0.00         0.00         0.00         5.33         0.00
192.168.1.4:/backup         8.74         0.00         0.00         0.00         8.74         0.00
192.168.1.8:/media          0.02         0.00         0.00         0.00         0.01         0.00

5. iostat – Display I/O data in MB/second

By default iostat, displays the device I/O statistics in Blocks. To change it to MB, use -m as shown below.

$ iostat -m
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db)       07/09/2011

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.68    0.00    0.52    2.03    0.00   91.76

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
sda             194.70         0.54         0.78    1327670    1935463
sda1            178.19         0.38         0.65     936370    1609060
sda2             16.51         0.16         0.13     391272     326402
sdb             371.27         0.46         0.52    1145240    1299425
sdb1            371.27         0.46         0.52    1145213    1299425
sdc             407.99         0.10         0.47     250666    1177259
sdc1            407.99         0.10         0.47     250639    1177259

6. iostat – Display I/O statistics only for a device

By default iostat displays I/O data for all the disks available in the system. To view statistics for a specific device (For example, /dev/sda), use the option -p as shown below.

$ iostat -p sda
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db)       07/09/2011

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.68    0.00    0.52    2.03    0.00   91.76

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             194.69      1096.51      1598.48 2719069928 3963829584
sda2            336.38        27.17        54.00   67365064  133905080
sda1            821.89         0.69       243.53    1720833  603892838

7. iostat – Display timestamp information

By default iostat displays only the current date. To display the current time, use the option -t as shown below.

$ iostat -t
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db)       07/09/2011

Time: 08:57:52 AM
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.68    0.00    0.52    2.03    0.00   91.76

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             194.69      1096.49      1598.45 2719070384 3963829704
sda1            178.18       773.32      1328.88 1917688474 3295357248
sda2             16.51       323.14       269.57  801326686  668472456
sdb             371.25       945.82      1073.16 2345452741 2661228872
sdb1            371.25       945.80      1073.16 2345397277 2661228872
sdc             407.97       207.02       972.27  513364233 2411030200
sdc1            407.97       207.00       972.27  513308769 2411030200

8. iostat – Display Extended status

Use option -x, which will displays extended disk I/O statistics information as shown below.

$ iostat -x
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db)       07/09/2011

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.68    0.00    0.52    2.03    0.00   91.76

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda              27.86    63.53 61.77 132.91  1096.46  1598.40    13.84     0.21    1.06   2.28  44.45
sda1              0.69    33.22 48.54 129.63   773.30  1328.84    11.80     1.39    7.82   2.28  40.57
sda2             27.16    30.32 13.23  3.28   323.13   269.56    35.90     0.55   32.96   3.44   5.68
sdb              39.15   215.16 202.20 169.04   945.80  1073.13     5.44     1.05    2.78   1.64  60.91
sdb1             39.15   215.16 202.20 169.04   945.77  1073.13     5.44     1.05    2.78   1.64  60.91
sdc               8.90     3.63 356.56 51.40   207.01   972.24     2.89     1.04    2.56   1.55  63.30
sdc1              8.90     3.63 356.55 51.40   206.99   972.24     2.89     1.04    2.56   1.55  63.30

To display extended information for a specific partition (For example, /dev/sda1), do the following.

$ iostat -x sda1
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db)       07/09/2011

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.68    0.00    0.52    2.03    0.00   91.76

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda1              0.69    33.21 48.54 129.62   773.23  1328.76    11.80     1.39    7.82   2.28  40.56

9. iostat – Execute Every x seconds (for y number of times)

To execute iostat every 2 seconds (until you press Ctl-C), do the following.

$ iostat 2
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db)       07/09/2011

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.68    0.00    0.52    2.03    0.00   91.76

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             194.67      1096.39      1598.33 2719070584 3963891256
sda1            178.16       773.26      1328.79 1917688482 3295418672
sda2             16.51       323.11       269.54  801326878  668472584
sdb             371.22       945.74      1073.08 2345454041 2661251200
sdb1            371.22       945.72      1073.08 2345398577 2661251200
sdc             407.93       207.00       972.19  513366813 2411036564
sdc1            407.93       206.98       972.19  513311349 2411036564
..

To execute every 2 seconds for a total of 3 times, do the following.

$ iostat 2 3

10. iostat – Display LVM statistic (and version)

To display the LVM statistics use option -N as shown below.

$ iostat -N

To display the version of iostat, use -V. This will really display the version information of sysstat, as iostat is part of sysstat package.

$ iostat -V
sysstat version 7.0.2
(C) Sebastien Godard

VMSTAT EXAMPLES

11. vmstat – Basic example

vmstat by default will display the memory usage (including swap) as shown below.

$ vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0 305416 260688  29160 2356920    2    2     4     1    0    0  6  1 92  2  0

vmstat output contains the following fields:

  • Procs – r: Total number of processes waiting to run
  • Procs – b: Total number of busy processes
  • Memory – swpd: Used virtual memory
  • Memory – free: Free virtual memory
  • Memory – buff: Memory used as buffers
  • Memory – cache: Memory used as cache.
  • Swap – si: Memory swapped from disk (for every second)
  • Swap – so: Memory swapped to disk (for every second)
  • IO – bi: Blocks in. i.e blocks received from device (for every second)
  • IO – bo: Blocks out. i.e blocks sent to the device (for every second)
  • System – in: Interrupts per second
  • System – cs: Context switches
  • CPU – us, sy, id, wa, st: CPU user time, system time, idle time, wait time

12. vmstat – Display active and inactive memory

By default vmstat doesn’t display this information. Use option -a, to display active and inactive memory information as shown below.

 
$ vmstat -a
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free  inact active   si   so    bi    bo   in   cs us sy id wa st
 0  0 305416 253820 1052680 2688928    2    2     4     1    0    0  6  1 92  2  0

13. vmstat – Display number of forks since last boot

This displays all the fork system calls made by the system since the last boot. This displays all fork, vfork, and clone system call counts.

$ vmstat -f
     81651975 forks

14. vmstat – Execute Every x seconds (for y number of times)

To execute every 2 seconds, do the following. You have to press Ctrl-C to stop this.

$ vmstat 2
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0      0 537144 182736 6789320    0    0     0     0    1    1  0  0 100  0  0	
 0  0      0 537004 182736 6789320    0    0     0     0   50   32  0  0 100  0  0	
..

To execute every 2 seconds for 10 times, do the following. You don’t need to press Ctrl-C in this case. After executing 10 times, it will stop automatically.

$ vmstat 2 10
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0      0 537144 182736 6789320    0    0     0     0    1    1  0  0 100  0  0	
 0  0      0 537004 182736 6789320    0    0     0     0   50   32  0  0 100  0  0	
..

15. vmstat – Display timestamp

When you use vmstat to monitor the memory usage repeately, it would be nice to see the timestap along with every line item. Use option -t to display the time stamp as shown below.

$ vmstat -t 1 100
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ ---timestamp---
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 3608728 148368 3898200    0    0     0     0    1    1  0  0 100  0  0     2011-07-09 21:16:28 PDT
 0  0      0 3608728 148368 3898200    0    0     0     0   60   15  0  0 100  0  0     2011-07-09 21:16:29 PDT
 0  0      0 3608712 148368 3898200    0    0     0     0   32   28  0  0 100  0  0     2011-07-09 21:16:30 PDT

For me, the timestamp option worked in the following version.

$ vmstat -V
procps version 3.2.8

Note: If you use a older version of vmstat, option -t might not be available. In that case, use the method we suggested earlier to display timestamp in vmstat output.

16. vmstat – Display slab info

Use option -m, to display the slab info as shown below.

$ vmstat -m
Cache                       Num  Total   Size  Pages
fib6_nodes                    5    113     32    113
ip6_dst_cache                 4     15    256     15
ndisc_cache                   1     15    256     15
RAWv6                         7     10    768      5
UDPv6                         0      0    640      6
tw_sock_TCPv6                 0      0    128     30
...

17. vmstat – Display statistics in a table format

Instead of displays the values in the record format, you can display the output of vmstat in table format using option -s as shown below.

$ vmstat -s
      4149928  total memory
      3864824  used memory
      2606664  active memory
      1098180  inactive memory
       285104  free memory
        19264  buffer memory
      2326692  swap cache
      4192956  total swap
       274872  used swap
      3918084  free swap
   1032454000 non-nice user cpu ticks
        14568 nice user cpu ticks
     89482270 system cpu ticks
  16674327143 idle cpu ticks
    368965706 IO-wait cpu ticks
      1180468 IRQ cpu ticks
..

18. vmstat – Display disk statistics

Use option -d to display the disk statistics as shown below. This displays the reads, writes, and I/O statistics of the disk.

$ vmstat -d
disk- ------------reads------------ ------------writes----------- -----IO------
       total merged sectors      ms  total merged sectors      ms    cur    sec
sda   153189971 69093708 2719150864 737822879 329617713 157559204 3965687592 4068577985      0 1102243
sdb   501426305 97099356 2345472425 731613156 419220973 533565961 2661869460 1825174087      0 1510434
sdc   884213459 22078974 513390701 452540172 127474901 8993357 2411187300 2133226954      0 1569758

19. vmstat – Increase the width of the display

The default output without increasing the width is shown below.

$ vmstat 1 3
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 3608688 148368 3898204    0    0     0     0    1    1  0  0 100  0  0
 0  0      0 3608804 148368 3898204    0    0     0     0   72   30  0  0 100  0  0
 0  0      0 3608804 148368 3898204    0    0     0     0   60   27  0  0 100  0  0

Use option -w to increase the width of the output columns as shown below. This give better readability.

$ vmstat -w 1 3
procs -------------------memory------------------ ---swap-- -----io---- --system-- -----cpu-------
 r  b       swpd       free       buff      cache   si   so    bi    bo   in   cs  us sy  id wa st
 0  0          0    3608712     148368    3898204    0    0     0     0    1    1   0  0 100  0  0
 0  0          0    3608712     148368    3898204    0    0     0     0   93   23   0  0 100  0  0
 0  0          0    3608696     148368    3898204    0    0     0     0   35   34   0  0 100  0  0

20. vmstat – Display statistics for a partition

To display the disk I/O statistics of a specific disk partition use option -p as shown below.

$ vmstat -p sdb1
sdb1          reads   read sectors  writes    requested writes
           501423248 2345417917  419221612 2661885948

21. vmstat – Display in MB

By default vmstat displays the memory information in kb. To disply in MB, use the option “-S m” as shown below.

$ vmstat -S m
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0    281    288     19   2386    0    0     4     1    0    0  6  1 92  2  0

MPSTAT EXAMPLES

22. mpstat – Display basic info

By default mpstat displays CPU statistics as shown below.

$ mpstat
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db)       07/09/2011

10:25:32 PM  CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   %idle    intr/s
10:25:32 PM  all    5.68    0.00    0.49    2.03    0.01    0.02    0.00   91.77    146.55

23. mpstat – Display all information

Option -A, displays all the information that can be displayed by the mpstat command as shown below. This is really equalivalent to “mpstat -I ALL -u -P ALL” command.

$ mpstat -A
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db)       07/09/2011      _x86_64_        (4 CPU)

10:26:34 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
10:26:34 PM  all    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00   99.99
10:26:34 PM    0    0.01    0.00    0.01    0.01    0.00    0.00    0.00    0.00   99.98
10:26:34 PM    1    0.00    0.00    0.01    0.00    0.00    0.00    0.00    0.00   99.98
10:26:34 PM    2    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:26:34 PM    3    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00

10:26:34 PM  CPU    intr/s
10:26:34 PM  all     36.51
10:26:34 PM    0      0.00
10:26:34 PM    1      0.00
10:26:34 PM    2      0.04
10:26:34 PM    3      0.00

10:26:34 PM  CPU     0/s     1/s     8/s     9/s    12/s    14/s    15/s    16/s    19/s    20/s    21/s    33/s   NMI/s   LOC/s   SPU/s   PMI/s   PND/s   RES/s   CAL/s   TLB/s   TRM/s   THR/s   MCE/s   MCP/s   ERR/s   MIS/s
10:26:34 PM    0    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    7.47    0.00    0.00    0.00    0.00    0.02    0.00    0.00    0.00    0.00    0.00    0.00    0.00
10:26:34 PM    1    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    4.90    0.00    0.00    0.00    0.00    0.03    0.00    0.00    0.00    0.00    0.00    0.00    0.00
10:26:34 PM    2    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.04    0.00    0.00    0.00    0.00    0.00    3.32    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
10:26:34 PM    3    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    4.17    0.00    0.00    0.00    0.00    0.01    0.00    0.00    0.00    0.00    0.00    0.00    0.00

24. mpstat – Display CPU statistics of individual CPU (or) Core

Option -P ALL, displays all the individual CPUs (or Cores) along with its statistics as shown below.

$ mpstat -P ALL
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db)       07/09/2011      _x86_64_        (4 CPU)

10:28:04 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
10:28:04 PM  all    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00   99.99
10:28:04 PM    0    0.01    0.00    0.01    0.01    0.00    0.00    0.00    0.00   99.98
10:28:04 PM    1    0.00    0.00    0.01    0.00    0.00    0.00    0.00    0.00   99.98
10:28:04 PM    2    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:28:04 PM    3    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00

To display statistics information of a particular CPU (or core), use option -P as shown below.

$ mpstat -P 0
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db)       07/09/2011      _x86_64_        (8 CPU)

10:28:53 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
10:28:53 PM    0    0.01    0.00    0.01    0.01    0.00    0.00    0.00    0.00   99.98

$ mpstat -P 1
Linux 2.6.32-100.28.5.el6.x86_64 (dev-db)       07/09/2011      _x86_64_        (8 CPU)

10:28:55 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
10:28:55 PM    1    0.00    0.00    0.01    0.00    0.00    0.00    0.00    0.00   99.98

Finally, as we mentioned earlier mpstat is part of the sysstat package. When you do mpstat -V, it will really display the version number of the systat package as shown below.

$ mpstat -V
sysstat version 9.0.4
(C) Sebastien Godard (sysstat  orange.fr)

Previous articles in the Linux performance monitoring and tuning series: