AMD EPYC 7003 Milan Workload Profile NIC Throughput Intensive

When running AMD EPYC 7003 Milan based servers in the standard configuration, you may experience brief drops in network data throughput. The workload profile setting NIC Throughput Intensive in the BIOS can remedy this. This setting deactivates the dynamic adjustment of the Infinity Fabric P-States and sets further parameters. This setting is required, for example, when operating Broadcom P225P or NVIDIA ConntectX-5 (MCX512A-ACAT) network cards.

ASUS RS700A-E11-RS12U based server: Product not available anymore.

The Workload Profile NIC Throughput Intensive enables high network data throughput.

Problem description

When operating multiple ASUS RS700A-E11-RS12U servers in a switched 25 Gigabit network, there are repeated drops in data throughput:

# iperf -m -i 1 -f G -t 60 -c 192.168.10.161
------------------------------------------------------------
Client connecting to 192.168.10.161, TCP port 5001
TCP window size:  325 KByte (default)
------------------------------------------------------------
[  1] local 192.168.10.160 port 45634 connected with 192.168.10.161 port 5001 (MSS=8960)
[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-1.0000 sec  2.79 GBytes  2.79 GBytes/sec
[  1] 1.0000-2.0000 sec  2.68 GBytes  2.68 GBytes/sec
[...]
[  1] 18.0000-19.0000 sec  2.86 GBytes  2.86 GBytes/sec
[  1] 19.0000-20.0000 sec  0.997 GBytes  0.997 GBytes/sec
[  1] 20.0000-21.0000 sec  0.166 GBytes  0.166 GBytes/sec
[  1] 21.0000-22.0000 sec  0.152 GBytes  0.152 GBytes/sec
[  1] 22.0000-23.0000 sec  0.117 GBytes  0.117 GBytes/sec
[  1] 23.0000-24.0000 sec  0.319 GBytes  0.319 GBytes/sec
[  1] 24.0000-25.0000 sec  2.31 GBytes  2.31 GBytes/sec
[...]

If two servers are wired directly (without a switch), these problems do not occur.

Test setup

• Server (4 pieces):
  • ASUS RS700A-E11-RS12U with BIOS 1201 (Release Date 09/07/2022, Milan Agesa code 1.0.0.9)
  • 2x AMD EPYC 7313 (3.00 GHz, 16 core, 128 MB)
  • 512 GB (8x 64GB) ECC Reg DDR4 2933 RAM
  • 2x 25 Gigabit Broadcom P225P or NVIDIA ConnectX-5 MCX512A-ACAT (in tests with NVIDIA ConnectX-6 MCX621102AN-ADAT no problems occurred)
  • Operating system: Ubuntu 22.04 LTS with Linux kernel 5.15
• Switch:
  • MikroTik CRS518-16XS-2XQ-RM with RouterOS 7.8 (Build Time Feb/24/2023 09:03:00).

Solution

To achieve consistently stable network throughput performance, select the following settings in the BIOS:

• 

  Go to the menu item Advanced in the BIOS.

• 

  Choose AMD CBS.

• 

  Go to the menu item Workload Tuning and choose Workload Profile.

• 

  Choose NIC Throughput Intensive.

• 

  After the setting is set, select [F10] to restart the server.

The Workload Profile NIC Throughput Intensive disables dynamic tuning of Infinity Fabric P-States through APBDIS=1 and sets them permanently to P0.

Enabling the workload profile NIC Throughput Intensive sets the following parameters, among others:

• xGMI Force Link Width = x16
• APBDIS = 1
• Fixed SOC P-State = P0
• DF C-States = Enabled
• Determinism Slider = Power
• IOMMU = Disabled (Linux)

References

Author: Werner Fischer Werner Fischer, working in the Knowledge Transfer team at Thomas-Krenn, completed his studies of Computer and Media Security at FH Hagenberg in Austria. He is a regular speaker at many conferences like LinuxTag, OSMC, OSDC, LinuxCon, and author for various IT magazines. In his spare time he enjoys playing the piano and training for a good result at the annual Linz marathon relay.