wikiQPLIB: bnxt re is fw stalled: FW STALL detected
Boot times may be extended when using Ubuntu 24.04 LTS (Linux Kernel 6.8) on servers with Supermicro H12SSL-NT mainboard (with onboard Broadcom network cards BCM57416). This is caused by a problem in the bnxt_re0 module.
Problem
Server startup takes longer than usual and the following entries appear in the system log:
[ 308.644788] bnxt_en 0000:45:00.0: QPLIB: bnxt_re_is_fw_stalled: FW STALL Detected. cmdq[0xe]=0x3 waited (137918 > 100000) msec active 1 [ 308.644801] bnxt_en 0000:45:00.0 bnxt_re0: Failed to modify HW QP [ 308.645509] infiniband bnxt_re0: Couldn't change QP1 state to INIT: -110 [ 308.646160] infiniband bnxt_re0: Couldn't start port [ 308.646578] bnxt_en 0000:45:00.0 bnxt_re0: Failed to destroy HW QP [ 308.647557] ------------[ cut here ]------------ [...] [ 308.649008] ---[ end trace 0000000000000000 ]--- [ 308.649016] bnxt_en 0000:45:00.0 bnxt_re0: Free MW failed: 0xffffff92 [ 308.649843] infiniband bnxt_re0: Couldn't open port 1 [ 308.651052] infiniband bnxt_re0: Device registered with IB successfully
Cause
The delay is caused by problems with the bnxt_re0 module which is usually used for RDMA over Converged Ethernet (RoCE).[1]
The exact cause is probably a change of the infiniband driver.[2][3]
Solution
We recommend to not load the module bnxt_re automatically, as the RoCE functionality is generally not required anyway.
This can be performed easily by the following configuration:
echo "blacklist bnxt_re" >> /etc/modprobe.d/blacklist-bnxt_re.conf update-initramfs -u
Restart the device afterwards.
References
- ↑ Validating RDMA over Converged Ethernet (RoCE) Network on Linux (techdocs.broadcom.com)
- ↑ The Broadcom 'bnxt' Ethernet driver and RDMA (in Ubuntu 24.04) (utcc.utoronto.ca/~cks/space/blog, 09.08.2024)
- ↑ (TUTORIAL) Broadcom NICs down after PVE 8.2 (Kernel 6.8) - Comment #13 (forum.proxmox.com)
More information
 |
Author: Werner Fischer Werner Fischer, working in the Knowledge Transfer team at Thomas-Krenn, completed his studies of Computer and Media Security at FH Hagenberg in Austria. He is a regular speaker at many conferences like LinuxTag, OSMC, OSDC, LinuxCon, and author for various IT magazines. In his spare time he enjoys playing the piano and training for a good result at the annual Linz marathon relay.
|
 |
Translator: Alina Ranzinger Alina has been working at Thomas-Krenn.AG since 2024. After her training as multilingual business assistant, she got her job as assistant of the Product Management and is responsible for the translation of texts and for the organisation of the department.
|
