wikiLinkspeed configuration of Broadcom network cards
When integrating Broadcom network cards in Linux distributions, the linkspeed may differ from the expected value. This value can be configured later with the help of firmware utility bnxtnvm.
This article describes the verification and configuration of the linkspeeds with the help of bnxtnvm/NICCLI.
Before you start, follow this article for the Activation of maintenance mode in Proxmox VE.
Verification of linkspeed
Due to the mixed operation of different linkspeeds, individual ports may be out of service. In a production environment, this manifests itself as missing connections and/or reduced network speeds, for example, when operating a server cluster with mesh cabling. All transceivers of the configured network card must be also removed.
Hint: Please note the Compatibility of linkspeeds. Certain Broadcom cards support only certain or no mixed operations.
Here is an example for a defective issue: root@PMX1:~# ethtool enp67s0f0np0
Settings for enp67s0f0np0:
Supported ports: [ FIBRE ]
Supported link modes: 10000baseT/Full
25000baseCR/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: Yes
Supported FEC modes: RS BASER
Advertised link modes: 10000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Advertised FEC modes: RS BASER
Speed: 10000Mb/s
Duplex: Full
Auto-negotiation: on
Port: Direct Attach Copper
PHYAD: 0
Transceiver: internal
Supports Wake-on: g
Wake-on: d
Current message level: 0x00002081 (8321)
drv tx_err hw
Link detected: yes
Under "Advertised link modes", only the speed is displayed (10 Gb/s), although the network card could run on 25 Gb/s.
Linkspeed configuration with bnxtnvm
Hint: If you are using a mesh network, you may need to disconnect the direct cabling of the systems to be configured before beginning the configuration process. The other systems can remain connected to each other.
To reach the desired and full functionality of the network cards, you must force the network cards to use the desired speed using bnxtnvm.
bnxtnvm can be downloaded in the Thomas-Krenn download area for your Windows systems or use the following commands on your Linux systems:
wget https://www.thomaskrenn.com/redx/tools/mb_download.php/ct.YuuHGw/mid.y9b3b4ba2bf7ab3b8/bnxtnvm.zip &&
unzip bnxtnvm.zip &&
chmod +x bnxtnvm
Next, you'll need to enter a series of commands to adjust your system to the desired speed:
Reading configurable speeds
In the first step, the driver linkspeed IDs of the network card are required:
./bnxtnvm -dev=enp67s0f0np0 optionhelp=drv_link_speed
Issue:
d
Name : drv_link_speed
Description : Driver Link speed
Option Type : Multi Instance Type
Max Instance Indexes : 0 to 15
Valid values :
0 (Autoneg)
1 (1G)
2 (10G)
3 (25G)
4 (40G)
5 (50G)
6 (100G)
7 (200G_PAM4)
8 (50G_PAM4)
9 (100G_PAM4)
14 (5G)
15 (100M)
The linkspeed ID is required for the enforcement of the desired linkspeed.
Configuration of linkspeed
The linkspeed is now set with the help of the linkspeed ID:
./bnxtnvm -dev=enp67s0f0np0 setoption=drv_link_speed:0#3
./bnxtnvm -dev=enp67s0f0np0 setoption=firmware_link_speed_d0:0#3
./bnxtnvm -dev=enp67s0f0np0 setoption=firmware_link_speed_d3:0#3
Explanation of the commands:
| Interface-ID | Option | Port | Linkspeed-ID |
|---|---|---|---|
| enp67s0f0np0 | drv_link_speed | 0 | 3 |
| enp67s0f0np0 | firmware_link_speed_d0 | 0 | 3 |
| enp67s0f0np0 | firmware_link_speed_d3 | 0 | 3 |
The drv_link_speed, firmware_link_speed_d0 & firmware_link_speed_d3 have to be set for all ports.
The values correspond to the number of ports of the used network cards (for example 0, 1, 2 and 3 for a network card with 4 network ports).
With the linkspeed ID "3", a speed of 25 Gb/s is forced.
You will receive the following issue after performing the command:
root@PMX1:~# ./bnxtnvm -dev=enp67s0f0np0 setoption=drv_link_speed:0#3
drv_link_speed is set successfully
Please reboot the system to apply the configuration
After entering the commands for all ports, a reboot of the system is necessary.
Verification of changes
Perform the following command to verify the state:
./bnxtnvm -dev=enp67s0f0np0 device_info
The configuration has been performed correctly, when the FW image status outputs the value Operational:
root@PMX1:~# ./bnxtnvm -dev=enp67s0f0np0 device_info
Device Interface Name : enp67s0f0np0 MACAddress : bc:97:e1:da:92:a0 Base MACAddress : BC:97:E1:DA:92:A0 Device Serial Number : P425G214400143FV Chip Number : BCM57504 Part Number : BCM957504-P425G Description : Broadcom NetXtreme-E Quad-port 25Gb Ethernet PCIe Adapter PCI Vendor Id : 14e4 PCI Device Id : 1750 PCI Subsys Vendor Id : 14e4 PCI Subsys Device Id : 2100 PCI Device Name : 0000:01:00.0 Adapter Rev : 11 Active Package version : 224.1.102.0 Package version on NVM : 224.1.102.0 Firmware version : 224.0.159.0 Active NVM config version : 0.0.34 NVM config version : 0.0.34 Firmware Reset Counter : 0 Error Recovery Counter : 0 Crash Dump Timestamp : N/A Reboot Required : No Secure Boot : Enabled Secure Firmware Update : Enabled FW Image Status : Operational Crash Dump Available in DDR : False
If this is not the case, the firmware of the network card must be synchronized additionally.
Firmware sync
Synchronizing the firmware will apply the entered link speed:
./bnxtnvm -dev=enp67s0f0np0 fw_sync
Control with the help of ethtool
The successful configuration can now be verified with the ethtool:root@PMX1:~# ethtool enp67f0np0
Settings for enp67f0np0:
Supported ports: [ FIBRE ]
Supported link modes: 10000baseT/Full
25000baseCR/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: Yes
Supported FEC modes: RS BASER
Advertised link modes: 25000baseCR/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Advertised FEC modes: RS BASER
Speed: 25000Mb/s
Duplex: Full
Auto-negotiation: on
Port: Direct Attach Copper
PHYAD: 0
Transceiver: internal
Supports Wake-on: g
Wake-on: d
Current message level: 0x00002081 (8321)
drv tx_err hw
Link detected: yes
Finally, you can disable maintenance mode again and, if necessary, repeat this procedure for the remaining nodes.
Linkspeed configuration with NICCLI
Hint: In case of a mesh cabling, it may be necessary that you disconnect the direct cabling of the systems to be configured before beginning the configuration. The other systems can remain wired to each other. You also have to remove all transceivers of the configured network card.
To ensure that the network cards function as intended and at full capacity, you must use NICCLI to force them to operate at the desired speed.
NICCLI installation
To install NICCLI, follow these instructions: Installation NICCLI
First, perform the following command to find out the assignment of the interface names and PCI-IDs:
root@PMX2:~# lshw -c network -businfo Bus info Device Class Description pci@0000:01:00.0 enp1s0f0np0 network BCM57508 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200Gb Ethernet pci@0000:01:00.1 enp1s0f1np1 network BCM57508 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200Gb Ethernet pci@0000:06:00.0 enp6s0f0 network I350 Gigabit Network Connection pci@0000:06:00.1 enp6s0f1 network I350 Gigabit Network Connection pci@0000:41:00.0 enp65s0f0np0 network BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller pci@0000:41:00.1 enp65s0f1np1 network BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller pci@0000:81:00.0 ens17f0np0 network BCM57504 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200Gb Ethernet pci@0000:81:00.1 ens17f1np1 network BCM57504 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200Gb Ethernet pci@0000:81:00.2 ens17f2np2 network BCM57504 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200Gb Ethernet pci@0000:81:00.3 ens17f3np3 network BCM57504 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200Gb Ethernet usb@1:1.3 enx42a65ce855c5 network Ethernet interface
Next, enter "niccli" in the CLI. After this, you have to select a port index. Select the index here that has the same PCI ID as the one selected above:
root@PMX2:~# niccli
/opt/niccli/niccli.x86_64: /lib/x86_64-linux-gnu/libnl-3.so.200: no version information available (required by /opt/niccli/niccli.x86_64)
-------------------------------------------------------------------------------
Scrutiny NIC CLI v229.0.150.0 - Broadcom Inc. (c) 2024 (Bld-79.52.33.102.16.0)
-------------------------------------------------------------------------------
BoardId MAC Address FwVersion PCIAddr Type Mode
1) BCM57508 84:16:0C:6F:16:90 226.0.145.1 0000:01:00.0 NIC PCI
2) BCM57508 84:16:0C:6F:16:90 226.0.145.1 0000:01:00.1 NIC PCI
3) BCM57416 00:62:0B:31:4A:F0 226.0.145.0 0000:41:00.0 NIC PCI
4) BCM57416 00:62:0B:31:4A:F1 226.0.145.0 0000:41:00.1 NIC PCI
5) BCM57504 00:62:0B:6F:00:8C 226.0.145.1 0000:81:00.0 NIC PCI
6) BCM57504 00:62:0B:6F:00:8C 226.0.145.1 0000:81:00.1 NIC PCI
7) BCM57504 00:62:0B:6F:00:8E 226.0.145.1 0000:81:00.2 NIC PCI
8) BCM57504 00:62:0B:6F:00:8E 226.0.145.1 0000:81:00.3 NIC PCI
Enter the target index to connect with : 5
Reading out configurable speeds
Now, you can read out the linkspeed ID for the desired port speed:
BCM57504> setoption -name firmware_link_speed_d0 -value ? -scope 0
ERROR: Invalid syntax.
Name : firmware_link_speed_d0
Description : This per-port option configures the speed setting when the device is in D0 mode.
The default speed is set to Autoneg. A 10GBaseT board only allows Autoneg.
Option Type : Multi Instance Type
Instance Indexes : 0 to 15
Valid values :
0 (Autoneg)
1 (1G)
2 (10G)
3 (25G)
4 (40G)
5 (50G)
6 (100G)
7 (200G_PAM4)
8 (50G_PAM4)
9 (100G_PAM4)
14 (2_5G)
15 (100M)
BCM57504>
The linkspeed ID is required for the enforcement of the desired linkspeed.
Configuration of the linkspeed
This linkspeed ID must be entered in the following command under "-value ":
BCM57504> setoption -name firmware_link_speed_d0 -value 3 -scope 0 BCM57504> setoption -name firmware_link_speed_d0 -value 3 -scope 1 BCM57504> setoption -name firmware_link_speed_d0 -value 3 -scope 2 BCM57504> setoption -name firmware_link_speed_d0 -value 3 -scope 3 firmware_link_speed_d0 is set successfully Please reboot the system to apply the configuration
The same must be repeated for "firmware_link_speed_d3":
BCM57504> setoption -name firmware_link_speed_d3 -value 3 -scope 0 BCM57504> setoption -name firmware_link_speed_d3 -value 3 -scope 1 BCM57504> setoption -name firmware_link_speed_d3 -value 3 -scope 2 BCM57504> setoption -name firmware_link_speed_d3 -value 3 -scope 3
Explanation of command:
| Option | Linkspeed-ID (value) | Port (scope) |
|---|---|---|
| firmware_link_speed_d0 | 3 | 0-3 |
| firmware_link_speed_d3 | 3 | 0-3 |
The firmware_link_speed_d0 & firmware_link_speed_d3 options must be set for all ports.
The values correspond to the number of ports of the used network card (for example 0, 1, 2 and 3 for a network card with 4 network ports).
With the linkspeed ID "3", a speed of 25 Gb/s is forced.
After entering the commands for all ports, a reboot of the system is necessary.
Verification of changes
Enter "niccli" again in the CLI after the reboot and perform the following command:
BCM57504> device_info Interface Name : ens17f0np0 MAC Address : 00:62:0B:6F:00:8C Base MAC Address : 00:62:0B:6F:00:8C Serial Number : N425G224200E5NFG Part Number : BCM957504-N425G PCI Address : 0000:81:00.0 Chip Number : BCM57504 Chip Name : THOR Description : Broadcom NetXtreme E-Series Quad-port 25Gb SFP28 OCP 3.0 Ethernet Adapter Active Package Name : 226.1.107.1 Firmware Name : PRIMATE_FW Firmware Version : 226.0.145.1 RoCE Firmware Version : 226.0.145.0 HWRM Interface Spec : 1.10.2 Kong mailbox channel : Not Applicable Active Package Version : 226.1.107.1 Package Version on NVM : 226.1.107.1 PCI Device ID : 0x1751 PCI Vendor ID : 0x14E4 PCI Revision ID : 0x11 PCI Component ID : 0x1751 PCI Subsys Device ID : 0x5425 PCI Subsys Vendor ID : 0x14E4 Active NVM config version : 0.0.37 NVM config version : 0.0.37 Reboot Required : No Firmware Reset Counter : 0 Error Recovery Counter : 0 Crash Dump Timestamp : Not Available Secure Boot : Enabled Secure Firmware Update : Enabled FW Image Status : Operational Crash Dump Available in DDR : No
Verify, whether the value is set to "Operational". If this is not the case, you have to perform a firmware sync. Otherwise, the configuration has been successful.
Firmware sync
Synchronizing the firmware will apply the entered link speed:
BCM57504> fw_sync
Control with the help of ethtool
Now, you can verify the configuration of the network card with ethtool:
root@PMX2:~# ethtool ens17f0np0
Settings for ens17f0np0:
Supported ports: [ FIBRE ]
Supported link modes: 10000baseT/Full
10000baseKX4/Full
10000baseKR/Full
25000baseCR/Full
25000baseKR/Full
25000baseSR/Full
10000baseCR/Full
10000baseSR/Full
10000baseLR/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: Yes
Supported FEC modes: RS BASER
Advertised link modes: 25000baseCR/Full
Advertised pause frame use: No
Advertised auto-negotiation: No
Advertised FEC modes: Not reported
Speed: 25000Mb/s
Duplex: Full
Auto-negotiation: off
Port: FIBRE
PHYAD: 0
Transceiver: internal
Supports Wake-on: g
Wake-on: d
Current message level: 0x00002081 (8321)
drv tx_err hw
Link detected: no
Finally, you can disable maintenance mode again and, if necessary, repeat this procedure for the remaining nodes.
 |
Author: Niklas Pauli Niklas Pauli works in the product management team at Thomas-Krenn. He completed his general university entrance qualification at Freyung High School and then his training as an IT specialist for system integration at Thomas-Krenn.AG. After completing his training, he continued to work in the areas of Proxmox VE incl. Ceph, backup (Proxmox, Veeam, SEP). He is also a regular writer for the Thomas-Krenn-Wiki.
|
 |
Translator: Alina Ranzinger Alina has been working at Thomas-Krenn.AG since 2024. After her training as multilingual business assistant, she got her job as assistant of the Product Management and is responsible for the translation of texts and for the organisation of the department.
|
