PCIe link lost

From Thomas-Krenn-Wiki
Jump to navigation Jump to search

On Linux systems, when using Intel I350 network chips, messages such as PCIe link lost may appear, followed by igb 0000:82:00.0 enp130s0f0: malformed Tx packet detected and dropped, LVMMC:0xffffffff. Therefore, the affected network interface does not work. As a solution, the Linux kernel parameter pcie_aspm.policy=performance or an updated BIOS with deactivated ASPM can be used.

Affected hardware and software

We have encountered this issue in isolated cases after several weeks of testing with the following components:

Problem

Example 21.6.2022: 

Jun 21 00:53:43 PMX2 kernel: [568234.890225] igb 0000:82:00.0 enp130s0f0: PCIe link lost
Jun 21 00:53:43 PMX2 kernel: [568234.890714] ------------[ cut here ]------------
Jun 21 00:53:43 PMX2 kernel: [568234.891150] igb: Failed to read reg 0x40e8!
Jun 21 00:53:43 PMX2 kernel: [568234.891611] WARNING: CPU: 15 PID: 1607760 at drivers/net/ethernet/intel/igb/igb_main.c:747 igb_rd32.cold+0x3a/0x46 [igb]
Jun 21 00:53:43 PMX2 kernel: [568234.892059] Modules linked in: sch_ingress ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter sctp ip6_udp_tunnel 
udp_tunnel nf_tables bonding tls openvswitch nsh nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 softdog nfnetlink_log nfnetlink ipmi_ssif intel_rapl_msr intel_rapl_common amd64_edac edac_
mce_amd kvm_amd ast drm_vram_helper drm_ttm_helper kvm ttm drm_kms_helper cec irqbypass crct10dif_pclmul rc_core ghash_clmulni_intel aesni_intel fb_sys_fops crypto_simd cryptd rapl wmi_bmof efi_pstore pc
spkr acpi_ipmi syscopyarea cdc_ether sysfillrect usbnet sysimgblt joydev input_leds mii ipmi_si ccp ptdma k10temp ipmi_devintf ipmi_msghandler mac_hid vhost_net vhost vhost_iotlb tap ib_iser rdma_cm iw_c

Jun 21 00:53:43 PMX2 kernel: [568234.892098]  zstd_compress hid_generic usbmouse usbkbd usbhid hid raid6_pq libcrc32c simplefb crc32_pclmul nvme nvme_core igb xhci_pci i2c_algo_bit ahci xhci_pci_renesas 
dca libahci xhci_hcd i2c_piix4 bnxt_en wmi
Jun 21 00:53:43 PMX2 kernel: [568234.896894] CPU: 15 PID: 1607760 Comm: nload Tainted: P           O      5.15.35-1-pve #1
Jun 21 00:53:43 PMX2 kernel: [568234.897392] Hardware name: ASUSTeK COMPUTER INC. RS500A-E11-RS12U/KMPA-U16 Series, BIOS 0901 12/03/2021
Jun 21 00:53:43 PMX2 kernel: [568234.897890] RIP: 0010:igb_rd32.cold+0x3a/0x46 [igb]
Jun 21 00:53:43 PMX2 kernel: [568234.898391] Code: c7 c6 94 23 3e c0 e8 ea e4 33 ed 48 8b bb 30 ff ff ff e8 5a 2c cc ec 84 c0 74 16 44 89 ee 48 c7 c7 60 30 3e c0 e8 b9 f2 2a ed <0f> 0b e9 b8 03 fe ff e9 
cf 03 fe ff 0f b6 d0 be 00 00 04 00 48 c7
Jun 21 00:53:43 PMX2 kernel: [568234.899403] RSP: 0018:ffffbc9e812d3a98 EFLAGS: 00010282
Jun 21 00:53:43 PMX2 kernel: [568234.899906] RAX: 0000000000000000 RBX: ffff947b663fced0 RCX: ffff949a0e1e0588
Jun 21 00:53:43 PMX2 kernel: [568234.900414] RDX: 00000000ffffffd8 RSI: 0000000000000027 RDI: ffff949a0e1e0580
Jun 21 00:53:43 PMX2 kernel: [568234.900915] RBP: ffffbc9e812d3ab0 R08: 0000000000000003 R09: 0000000000000001
Jun 21 00:53:43 PMX2 kernel: [568234.901411] R10: ffff947b7610c7c0 R11: ffffffffc0d410c0 R12: 00000000ffffffff
Jun 21 00:53:43 PMX2 kernel: [568234.901902] R13: 00000000000040e8 R14: 00000000000198e8 R15: 0000000003f1d080
Jun 21 00:53:43 PMX2 kernel: [568234.902389] FS:  00007f4cb056d740(0000) GS:ffff949a0e1c0000(0000) knlGS:0000000000000000
Jun 21 00:53:43 PMX2 kernel: [568234.902879] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 21 00:53:43 PMX2 kernel: [568234.903364] CR2: 00007f4f7bbf2670 CR3: 00000006414e8004 CR4: 0000000000770ee0
Jun 21 00:53:43 PMX2 kernel: [568234.903850] PKRU: 55555554
Jun 21 00:53:43 PMX2 kernel: [568234.904336] Call Trace:
Jun 21 00:53:43 PMX2 kernel: [568234.904808]  <TASK>
Jun 21 00:53:43 PMX2 kernel: [568234.905275]  igb_update_stats+0x4c0/0x880 [igb]
Jun 21 00:53:43 PMX2 kernel: [568234.905743]  igb_get_stats64+0x30/0x80 [igb]
Jun 21 00:53:43 PMX2 kernel: [568234.906205]  dev_get_stats+0x60/0xc0
Jun 21 00:53:43 PMX2 kernel: [568234.906663]  netstat_show.constprop.0+0x57/0xb0
Jun 21 00:53:43 PMX2 kernel: [568234.907117]  tx_dropped_show+0x16/0x20
Jun 21 00:53:43 PMX2 kernel: [568234.907564]  dev_attr_show+0x1d/0x40
Jun 21 00:53:43 PMX2 kernel: [568234.908014]  sysfs_kf_seq_show+0xa1/0x100
Jun 21 00:53:43 PMX2 kernel: [568234.908451]  kernfs_seq_show+0x27/0x30
Jun 21 00:53:43 PMX2 kernel: [568234.908882]  seq_read_iter+0x122/0x4b0
Jun 21 00:53:43 PMX2 kernel: [568234.909309]  ? aa_file_perm+0x11e/0x570
Jun 21 00:53:43 PMX2 kernel: [568234.909732]  kernfs_fop_read_iter+0x150/0x1b0
Jun 21 00:53:43 PMX2 kernel: [568234.910150]  new_sync_read+0x110/0x1a0
Jun 21 00:53:43 PMX2 kernel: [568234.910564]  vfs_read+0x100/0x1a0
Jun 21 00:53:43 PMX2 kernel: [568234.910970]  ksys_read+0x67/0xe0
Jun 21 00:53:43 PMX2 kernel: [568234.911368]  __x64_sys_read+0x1a/0x20
Jun 21 00:53:43 PMX2 kernel: [568234.911766]  do_syscall_64+0x5c/0xc0
Jun 21 00:53:43 PMX2 kernel: [568234.912170]  ? do_syscall_64+0x69/0xc0
Jun 21 00:53:43 PMX2 kernel: [568234.912560]  ? do_syscall_64+0x69/0xc0
Jun 21 00:53:43 PMX2 kernel: [568234.912944]  ? syscall_exit_to_user_mode+0x27/0x50
Jun 21 00:53:43 PMX2 kernel: [568234.913330]  ? __x64_sys_read+0x1a/0x20
Jun 21 00:53:43 PMX2 kernel: [568234.913706]  ? do_syscall_64+0x69/0xc0
Jun 21 00:53:43 PMX2 kernel: [568234.914073]  ? syscall_exit_to_user_mode+0x27/0x50
Jun 21 00:53:43 PMX2 kernel: [568234.914431]  ? do_syscall_64+0x69/0xc0
Jun 21 00:53:43 PMX2 kernel: [568234.914777]  ? __x64_sys_close+0x12/0x40
Jun 21 00:53:43 PMX2 kernel: [568234.915112]  ? do_syscall_64+0x69/0xc0
Jun 21 00:53:43 PMX2 kernel: [568234.915436]  ? do_syscall_64+0x69/0xc0
Jun 21 00:53:43 PMX2 kernel: [568234.915745]  ? do_syscall_64+0x69/0xc0
Jun 21 00:53:43 PMX2 kernel: [568234.916052]  ? asm_sysvec_apic_timer_interrupt+0xa/0x20
Jun 21 00:53:43 PMX2 kernel: [568234.916342]  entry_SYSCALL_64_after_hwframe+0x44/0xae
Jun 21 00:53:43 PMX2 kernel: [568234.916623] RIP: 0033:0x7f4cb0666e8e
Jun 21 00:53:43 PMX2 kernel: [568234.916890] Code: c0 e9 b6 fe ff ff 50 48 8d 3d 6e 18 0a 00 e8 89 e8 01 00 66 0f 1f 84 00 00 00 00 00 64 8b 04 25 18 00 00 00 85 c0 75 14 0f 05 <48> 3d 00 f0 ff ff 77 5a 
c3 66 0f 1f 84 00 00 00 00 00 48 83 ec 28
Jun 21 00:53:43 PMX2 kernel: [568234.917444] RSP: 002b:00007ffe81260f78 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
Jun 21 00:53:43 PMX2 kernel: [568234.917719] RAX: ffffffffffffffda RBX: 0000000000001fff RCX: 00007f4cb0666e8e
Jun 21 00:53:43 PMX2 kernel: [568234.917988] RDX: 0000000000001fff RSI: 000055e09cb3adf0 RDI: 0000000000000003
Jun 21 00:53:43 PMX2 kernel: [568234.918250] RBP: 000055e09cb3adf0 R08: 0000000000000000 R09: 00007f4cb0736be0
Jun 21 00:53:43 PMX2 kernel: [568234.918508] R10: 0000000001000000 R11: 0000000000000246 R12: 0000000000001fff
Jun 21 00:53:43 PMX2 kernel: [568234.918764] R13: 00007ffe81261118 R14: 000055e09bc21398 R15: 000055e09bc21370
Jun 21 00:53:43 PMX2 kernel: [568234.919020]  </TASK>
Jun 21 00:53:43 PMX2 kernel: [568234.919272] ---[ end trace e754faf722c4d59f ]---
Jun 21 00:53:43 PMX2 kernel: [568234.919529] igb 0000:82:00.0 enp130s0f0: malformed Tx packet detected and dropped, LVMMC:0xffffffff
Jun 21 00:53:43 PMX2 kernel: [568234.919586] igb 0000:82:00.1 enp130s0f1: PCIe link lost
Jun 21 00:53:43 PMX2 kernel: [568234.920381] ------------[ cut here ]------------
Jun 21 00:53:43 PMX2 kernel: [568234.920832] igb: Failed to read reg 0xc030!
Jun 21 00:53:43 PMX2 kernel: [568234.921292] WARNING: CPU: 47 PID: 1607760 at drivers/net/ethernet/intel/igb/igb_main.c:747 igb_rd32.cold+0x3a/0x46 [igb]
Jun 21 00:53:43 PMX2 kernel: [568234.921782] Modules linked in: sch_ingress ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter sctp ip6_udp_tunnel 
udp_tunnel nf_tables bonding tls openvswitch nsh nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 softdog nfnetlink_log nfnetlink ipmi_ssif intel_rapl_msr intel_rapl_common amd64_edac edac_
mce_amd kvm_amd ast drm_vram_helper drm_ttm_helper kvm ttm drm_kms_helper cec irqbypass crct10dif_pclmul rc_core ghash_clmulni_intel aesni_intel fb_sys_fops crypto_simd cryptd rapl wmi_bmof efi_pstore pc
spkr acpi_ipmi syscopyarea cdc_ether sysfillrect usbnet sysimgblt joydev input_leds mii ipmi_si ccp ptdma k10temp ipmi_devintf ipmi_msghandler mac_hid vhost_net vhost vhost_iotlb tap ib_iser rdma_cm iw_c
m ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi drm sunrpc ip_tables x_tables autofs4 zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) btrfs blak
e2b_generic xor

Beispiel 26.6.2022: 

[...]
Jun 26 18:46:36 PMX2 kernel: [1064607.600707] igb 0000:82:00.0 enp130s0f0: malformed Tx packet detected and dropped, LVMMC:0xffffffff
Jun 26 18:46:37 PMX2 kernel: [1064608.624690] igb 0000:82:00.1 enp130s0f1: malformed Tx packet detected and dropped, LVMMC:0xffffffff
[...]

Possible solutions

Updated BIOS

The new BIOS version 1301 (Milan, 2023/03/15) deactivates ASPM on all PCIe ports (Disable ASPM on all PCIe ports.) and resolves the problem.

Kernel parameter

Alternatively, as a workaround, the PCIe ASPM policy can be set to performance using the following kernel boot parameter:

sed -i '$ s/$/ pcie_aspm.policy=performance/' /etc/kernel/cmdline

On systemd-boot-systems, this command adds the parameter to the /etc/kernel/cmdline file. On Proxmox systems, a refresh of the Proxmox boot tool must be performed after changing the file: 

proxmox-boot-tool refresh

After a reboot, it can be verified if the parameters are active using 

cat /proc/cmdline

More information

References

Author: Werner Fischer

Werner Fischer, working in the Knowledge Transfer team at Thomas-Krenn, completed his studies of Computer and Media Security at FH Hagenberg in Austria. He is a regular speaker at many conferences like LinuxTag, OSMC, OSDC, LinuxCon, and author for various IT magazines. In his spare time he enjoys playing the piano and training for a good result at the annual Linz marathon relay.


Author: Jonas Sterr

Jonas Sterr has been working for Thomas-Krenn for several years. Originally employed as a trainee in technical support and then in hosting (formerly Filoo), Mr. Sterr now mainly deals with the topics of storage (SDS / Huawei / Netapp), virtualization (VMware, Proxmox, HyperV) and network (switches, firewalls) in product management at Thomas-Krenn.AG in Freyung.


Translator: Alina Ranzinger

Alina has been working at Thomas-Krenn.AG since 2024. After her training as multilingual business assistant, she got her job as assistant of the Product Management and is responsible for the translation of texts and for the organisation of the department.


Related articles

Linux Performance Measurements using vmstat
Show article
Random Reboots AMD EPYC server
Show article
Rdiff-backup Monitoring Plugin
Show article