Linux CPU performance measurements with mpstat

The mpstat Linux tool provides information on the capacity of the individual CPUs of a system.With multicore CPUs, each core is displayed as a CPU. For CPUs with hyperthreading, two CPUs are displayed per physical CPU or physical core. mpstat is usually included in the sysstat package in Linux distributions. (see for example Debian http://packages.debian.org/sysstat).

Basics of mpstat

mpstat is typically called up with the option mpstat -P ALL. This will list all CPUs individually. A call of mpstat -P ALL shows the average rate since the last booting:

wfischer@ubuntu-1204:~$ sudo mpstat -P ALL
Linux 3.2.0-93-generic (ubuntu-1204) 	11/11/2015 	_x86_64_	(4 CPU)

09:38:16 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
09:38:16 AM  all    1.94    0.09    0.57    6.95    0.00    0.04    0.00    0.00   90.42
09:38:16 AM    0    1.01    0.20    0.69    3.77    0.00    0.08    0.00    0.00   94.27
09:38:16 AM    1    3.70    0.08    0.67   10.18    0.00    0.07    0.00    0.00   85.30
09:38:16 AM    2    1.72    0.05    0.50   10.28    0.00    0.00    0.00    0.00   87.45
09:38:16 AM    3    1.32    0.04    0.41    3.57    0.00    0.00    0.00    0.00   94.66
wfischer@ubuntu-1204:~$

For the analysis of a certain period, two more parameters can be stated. At mpstat -P ALL 1 60, the current value is read out every second (and this 60 times which means one minute). The last entry shows the average values over the measurement period. You can write the output of mpstat -P ALL 1 60 into a file:

wfischer@ubuntu-1204:~$ sudo mpstat -P ALL 1 60 > mpstatanalyse20151111-1.txt
wfischer@ubuntu-1204:~$

Here is an excerpt of this file:

Linux 3.2.0-93-generic (ubuntu-1204)    11/11/2015      _x86_64_        (4 CPU)

09:35:57 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
09:35:58 AM  all    0.00    0.00    0.00   15.19    0.00    0.00    0.00    0.00   84.81
09:35:58 AM    0    0.00    0.00    0.00   18.37    0.00    0.00    0.00    0.00   81.63
09:35:58 AM    1    0.00    0.00    0.99   29.70    0.00    0.99    0.00    0.00   68.32
09:35:58 AM    2    0.00    0.00    1.00   11.00    0.00    0.00    0.00    0.00   88.00
09:35:58 AM    3    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00

[...]

09:36:56 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
09:36:57 AM  all    3.26    0.00    1.50    0.75    0.00    0.00    0.00    0.00   94.49
09:36:57 AM    0    6.06    0.00    0.00    0.00    0.00    0.00    0.00    0.00   93.94
09:36:57 AM    1    0.00    0.00    0.00    2.00    0.00    0.00    0.00    0.00   98.00
09:36:57 AM    2    4.95    0.00    5.94    1.98    0.00    0.00    0.00    0.00   87.13
09:36:57 AM    3    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00

Average:     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
Average:     all    0.48    0.00    0.70   15.71    0.00    0.04    0.00    0.00   83.07
Average:       0    0.66    0.00    1.19   16.44    0.00    0.12    0.00    0.00   81.60
Average:       1    0.29    0.00    0.44   28.94    0.00    0.07    0.00    0.00   70.27
Average:       2    0.53    0.00    0.65   15.27    0.00    0.00    0.00    0.00   83.54
Average:       3    0.42    0.00    0.50    2.27    0.00    0.00    0.00    0.00   96.81

Meaning of the individual values

(source man mpstat):

• CPU: Processor number. The keyword all indicates that statistics are calculated as averages among all processors.
• %usr: Show the percentage of CPU utilization that occurred while executing at the user level (application).
• %nice: Show the percentage of CPU utilization that occurred while executing at the user level with nice priority.
• %sys: Show the percentage of CPU utilization that occurred while executing at the system level (kernel). Note that this does not include time spent servicing hardware and software interrupts.
• %iowait: Show the percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O request.
• %irq: Show the percentage of time spent by the CPU or CPUs to service hardware interrupts.
• %soft: Show the percentage of time spent by the CPU or CPUs to service software interrupts.
• %steal: Show the percentage of time spent in involuntary wait by the virtual CPU or CPUs while the hypervisor was servicing another virtual processor.
• %guest: Show the percentage of time spent by the CPU or CPUs to run a virtual processor.
• %idle: Show the percentage of time that the CPU or CPUs were idle and the system did not have an outstanding disk I/O request.

Analysis

Capacity of the individual CPUs or cores

In a system with multiple CPUs or multiple cores, it makes sense to utilize them evenly whenever possible. If the average values of mpstat -P ALL show one CPU with a very high %user share, but other CPUs with a very low %user share, optimizing the configuration of the applications may increase the overall performance of the system. A data base, that only uses a single thread or process, can only utilize one core.

Period of analysis

As a mpstat -P ALL provides all average values since the last booting process (and thus also includes times with low load - such as at night and on weekends), for example, a ten-minute measurement during peak times can be performed using mpstat -P ALL 1 600 > /root/mpstatanalyse.txt. This also gives you average values for these ten minutes at the end of the output.

Author: Werner Fischer Werner Fischer, working in the Knowledge Transfer team at Thomas-Krenn, completed his studies of Computer and Media Security at FH Hagenberg in Austria. He is a regular speaker at many conferences like LinuxTag, OSMC, OSDC, LinuxCon, and author for various IT magazines. In his spare time he enjoys playing the piano and training for a good result at the annual Linz marathon relay.

Translator: Alina Ranzinger Alina has been working at Thomas-Krenn.AG since 2024. After her training as multilingual business assistant, she got her job as assistant of the Product Management and is responsible for the translation of texts and for the organisation of the department.