wikiOptimize memory performance of Intel Xeon Scalable systems
Intel Xeon Scalable Performance series processors have two built-in memory controllers that can control up to three channels each. Up to two DIMMs are possible per channel. In order to achieve the best possible overall performance, it makes sense to ensure a balanced configuration. In this article we show frequently used DIMM configurations.
Basics
Processors of the Intel Xeon Scalable Performance series are characterized by the following features on the memory side:
- 2 memory controllers per CPU
- 3 memory channels per memory controller (6 memory channels per CPU = 12 memory channels in a dual CPU system)
- 2 DIMMs per channel (maximum 12 DIMMs per CPU = 24 DIMMs in a dual CPU system)
Configurations
The following tables show the ideal configurations of the different RAM slots.
Important note on maximum memory bandwidth: The tables show measurements of a bandwidth test with STREAM Triad.[1] The highest possible memory bandwidth is particularly relevant in the HPC environment. In other application areas, the influence of memory bandwidth on overall performance is lower and depends on the respective application. Tests with the SPECint_rate_base2006, for example, show that even with a memory bandwidth of 35%, the SPEC benchmark achieves up to 90% performance.[2] RAM assemblies with fewer modules reduce the maximum achievable memory bandwidth, but also require less energy and offer good expandability for the future.
Single-CPU systems with 8 DIMM slots
The following table shows the possible DIMM configurations of a single CPU system with a Supermicro X11SPL-F motherboard. The maximum possible memory bandwidth can be achieved with a configuration of 6 DIMMs:
| Module size | 2 DIMMs | 4 DIMMs | 6 DIMMs | 8 DIMMs |
|---|---|---|---|---|
| 8 GB | 16 GB | 32 GB | 48 GB | (*) |
| 16 GB | 32 GB | 64 GB | 96 GB | (*) |
| 32 GB | 64 GB | 128 GB | 192 GB | (*) |
| 64 GB | 128 GB | 256 GB | 384 GB | 512 GB(**) |
| Assembly | ||||
| Max. memory bandwidth[3][2] |
~ 35% | ~ 67 % | ~ 97 % | ~ 35 % |
- *) Configuration only recommended when upgrading an existing system, otherwise prefer DIMMs with higher capacity.
- **) Configuration recommended when maximum RAM capacity is required.
Dual CPU systems with 16 DIMM slots
The following table shows the possible DIMM configurations of a dual CPU system with a Supermicro X11DPi-NT motherboard. The maximum possible memory bandwidth can be achieved with a configuration of 12 DIMMs:
| Module size | 4 DIMMs (2 per CPU) |
8 DIMMs (4 per CPU) |
12 DIMMs (6 per CPU) |
16 DIMMs (8 per CPU) |
|---|---|---|---|---|
| 8 GB | 32 GB | 64 GB | 96 GB | (*) |
| 16 GB | 64 GB | 128 GB | 192 GB | (*) |
| 32 GB | 128 GB | 256 GB | 384 GB | (*) |
| 64 GB | 256 GB | 512 GB | 768 GB | 1.024 GB(**) |
| Assembly | ||||
| Max. memory bandwidth[3][2] |
~ 35% | ~ 67 % | ~ 97 % | ~ 34 % |
- *) Configuration only recommended when upgrading an existing system, otherwise prefer DIMMs with higher capacity.
- **) Configuration recommended when maximum RAM capacity is required.
Dual CPU systems with 24 DIMM slots
The following table shows the possible DIMM configurations of a dual CPU system with an ASUS Z11PP-D24 motherboard. The maximum possible memory bandwidth can be achieved with memory modules with up to 2.666 MHz in a configuration with 24 DIMMs (with Cascade Lake the configuration with 2 Dimms-per-Channel leads to a reduction to 2.666 MHz when 2.933 MHz modules are used - when 2.933 MHz modules are used, the memory bandwidth is therefore higher with 12 DIMMs):
| Module size | 4 DIMMs (2 per CPU) |
8 DIMMs (4 per CPU) |
12 DIMMs (6 per CPU) |
16 DIMMs (8 per CPU) |
20 DIMMs (10 per CPU) |
24 DIMMS (12 per CPU) |
|---|---|---|---|---|---|---|
| 8 GB | 32 GB | 64 GB | 96 GB | 128 GB | 160 GB | 192 GB |
| 16 GB | 64 GB | 128 GB | 192 GB | 256 GB | 320 GB | 384 GB |
| 32 GB | 128 GB | 256 GB | 384 GB | 512 GB | 640 GB | 768 GB |
| 64 GB | 256 GB | 512 GB | 768 GB | 1.024 GB | 1.280 GB | 1.536 GB |
| Assembly | ||||||
| Max. memory bandwidth[3][2] |
~ 35% | ~ 67 % | ~ 97 % | ~ 68 % | ~ 67% | 100 % |
Further information
- Memory Confiuguration for X11 UP/DP/MP Motherboards (www.supermicro.com)
References
- ↑ STREAM: Sustainable Memory Bandwidth in High Performance Computers (www.cs.virginia.edu)
- ↑ 2.0 2.1 2.2 2.3 Memory Performance of Xeon scalable processor (Skylake-SP) based Systems (sp.ts.fujitsu.com)
- ↑ 3.0 3.1 3.2 Maximizing System x and ThinkServer Performance with a Balanced Memory Configuration (lenovopress.com)
 |
Author: Werner Fischer Werner Fischer, working in the Knowledge Transfer team at Thomas-Krenn, completed his studies of Computer and Media Security at FH Hagenberg in Austria. He is a regular speaker at many conferences like LinuxTag, OSMC, OSDC, LinuxCon, and author for various IT magazines. In his spare time he enjoys playing the piano and training for a good result at the annual Linz marathon relay.
|
