Measuring I/O performance

In this article, we describe basics about Measuring I/O Performance. This topic is quite complex and extensive. We try to document the most important basic information without claiming to be exhaustive.

Basics

General basics

Ensure that tests are fully traceable. It is necessary that you document the whole test environment accurately.

On the hardware side, this includes, for example:

• mainboard including BIOS-version
• CPUs
• working memory
• (if used) RAID-controllers including firmware-version
• storage media (hard drives, SSDs, ...) including firmware-versions
• ...

Software-sided:

• BIOS configuration (it is best to select "Load Defaults" and then document any configuration changes)
• operating system with detailed information on the installation, software-version, etc.)
• (if used) exact configuration of the RAID-controllers (RAID-level, cache-configuration,...)
• testing tools with exact version release
• ...

Measuring SSDs and flash storage

• Same initial state: The writing performance of SSDs and flash storage may change with the filling level (see Solid State Drive - Schreibzugriffe. A nearly full written SSD has often a lower writing performance than an empty SSD that has been deleted completely by Secure Erase. When measuring performance, therefore, make sure that you either completely erase the SSDs beforehand using "Secure Erase" or first write a specific amount of data to them in a traceable manner.
• Same cache settings: When comparing multiple SSDs, make sure that the cache settings are as similar as possible.(see also Solid State Drive - DRAM buffer (Cache)).
• Test with random data: Some SSD controllers have integrated a data compression. Testing with zero data can lead to very high measuring values that cannot be reached with real data. It is best to test with random test data (see Solid State Drive - data compression).

Measuring hard disks

• Compare identical areas of hard disks: At a constant rotational speed, the data throughput at the outer edge of the disk platters is higher than at the inner edge. Since hard disks are written from the outside in, the throughput at the beginning of the data area is therefore higher than at the end of the data area of the hard disk. When comparing multiple hard disks, therefore, make sure to compare the same areas of the disks. The easiest way to test hard disks is directly at the beginning of the data area. If you prefer to test the end of the data area (i.e., the inner tracks of hard disks), start using the offset parameters of the performance tools, e.g., at 90% of the capacity of the respective hard disk.

  Example: Western Digital WD5002ABYS-01B1B0

  Start of data area

  • 111 MByte/s (tested with fio --filename=/dev/sdd --direct=1 --rw=read --bs=1m --size=5G --offset=0G --name=file1)
  • 116 MByte/s (tested with dd if=/dev/sdd of=/dev/null bs=1M count=5120 iflag=direct)

  End of data area

  • 62,8 MByte/s (tested with fio --filename=/dev/sdd --direct=1 --rw=read --bs=1m --size=5G --offset=460G --name=file1)
  • 65,8 MByte/s (tested with dd if=/dev/sdd of=/dev/null bs=1M count=5120 skip=471040 iflag=direct)

Note: If you require a high data transfer rate and I/O performance, it may be useful to use a larger hard disk only for the front area. If you require approximately 250 GByte, you can use a 1 TB hard disk, but only use the first 250 GByte. This increases the average data transfer rate and also the number of possible IOPS (= lower access time), as the hard disk arm no longer has to move so far inward.

• Same hard disk writing cache settings: If you compare multiple hard disks, please note the identical writing cache settings.
• No running SMART tests: Hard disks can execute self-tests on their own. These tests can be started, for example, with smartctl -t TEST. Such self tests influence the performance of parallel running I/O performance measurements. Please note therefore that no hard disk self-tests are running if you want to measure I/O performance.
• No running background operations as Pre-Scan from Seagate: Newer SAS hard disks from Seagate automatically perform a subsequent media scan when writing to a sector for the first time. This is intended to enable media defects to be detected at an early stage. The writing performance decreases by approximately 50 per cent. Therefore, describe such hard disks completely before testing or productive use (e.g., with dd). This completes the pre-scan and thus has no further negative performance effects. Note: These hard disks also perform a background media scan in parallel during IDLE operation.

Measuring RAID controllers

When performing measurements on hard drives or SSDs in a RAID array, please also note the following comments in addition to the above information:

• No running background processes: No background processes may be running on the RAID controller during performance measurements. RAID initialisations, RAID-verifies, etc. must be completed before.

Test cases

Throughput

Possible throughput rates are normally measured by throughput tests for sequential accesses (streaming I/O or sequential I/O).

The following is interesting for this:

• Reading throughput in MByte/sec
• Writing throughput in MByte/sec

Examples for throughput tests can be found here: fio - Testfälle.

IOPS

With IOPS tests (Input/Outputs per second), the number of possible in/output operations per second are measured. In general, the worst-case scenarios (Random Read IOPS and Random Write IOPS) are measured. These are randomly distributed accesses (random I/O), where each access transfers only a small amount of data.

IOPS can directly convert to the average access time of a hard disk. The average access time in seconds is the reciprocal of the IOPS. 71 IOPS correspond to approximately 0,01408 seconds = 14,08 ms average access time.

The average access time of hard disks consists of (see also Folie 9 of the presentation 'storage technologies in comparison'):

• seek time
• latency (duration of a half turn of the hard disk)
• controller overhead (command latency, can usually be disregarded)
• transfer time (can usually be disregarded, as it is quite low compared to seek time and latency)

Examples for IOPS-tests can be found here: fio - test cases.

More information on IOPS (I/Os per second) can also be found here:

• http://blogs.techrepublic.com.com/datacenter/?p=2182
• http://www.yellow-bricks.com/2009/12/23/iops/comment-page-1/
• http://en.wikipedia.org/wiki/IOPS

More information

• Testing virtual machine storage I/O performance (VMware Knowledge Base)

Author: Werner Fischer Werner Fischer, working in the Knowledge Transfer team at Thomas-Krenn, completed his studies of Computer and Media Security at FH Hagenberg in Austria. He is a regular speaker at many conferences like LinuxTag, OSMC, OSDC, LinuxCon, and author for various IT magazines. In his spare time he enjoys playing the piano and training for a good result at the annual Linz marathon relay.

Translator: Alina Ranzinger Alina has been working at Thomas-Krenn.AG since 2024. After her training as multilingual business assistant, she got her job as assistant of the Product Management and is responsible for the translation of texts and for the organisation of the department.