Hello,
I have a strange problem occurring with my new server which has the intel x520 network cards. This server runs Ubuntu KVM with virtual machines that collect netflow data- these device are virtual appliances. I noticed within the virtual machine that netflow datagrams. I first suspected resource constraints within KVM however after disabling all of the virtual appliances (except one) the packet drops were still occurring. I am able to determine the packet drops by simply taking a pcap and running it in a script with tshark. Each UDP datagram that contains the netflow data has a sequence number in the application layer header. I then ran a pcap on the host bridge interface and found that the issue is occurring outside the virtual machine. I then did a packet capture on physical interface to rule out that it was caused by the bridge and the issue is occurring there too. I then mirrored the interface to a separate server that has a similar 10gig network card and this device is able to capture each packet.
I suspect this is a network card offload /drive issue however I have not been successful in finding a solution. I tried upgrading the driver from 4.0.1 to 4.3.13 with no difference.
I am running out of options.
I tried disabling
for i in rx tx sg tso ufo gso gro lro; do sudo ethtool -K enp33s0f1 $i off; done
Tried increasing memory allocation to receive memory buffer:
cat /proc/sys/net/core/rmem_default
212992
cat /proc/sys/net/core/rmem_max
212992
and set the values to:
net.core.rmem_default = 16777216
net.core.rmem_max = 16777216
I noticed packets were incrementing the below counters in ethtool -S so i tried the below modprobe command but it did not help.
hw_rsc_aggregated: 4074
hw_rsc_flushed: 1837
modprobe ixgbe InterruptThrottleRate=off
server with issue:
lspci |grep Ethernet
0c:00.0 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)
0c:00.1 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)
21:00.0 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)
21:00.1 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)
62:00.0 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)
62:00.1 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)
63:00.0 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)
63:00.1 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)
I first tried with this below driver version.
version: | 4.0.1-k |
Then upgraded and tried the below driver version.
version: | 4.3.13 |
server without issue:
[em621d@shdw01vmh ~]$ lspci |grep Ethernet
0e:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
0e:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
version: | 3.6.7-k |
Traffic characteristics:
Under 150Mbps
Rate 10k pps
Packet size 1340 - 1372 bytes
ethtool -S enp33s0f1
NIC statistics:
rx_packets: 728170
tx_packets: 8
rx_bytes: 1007244560
tx_bytes: 648
rx_errors: 0
tx_errors: 0
rx_dropped: 0
tx_dropped: 0
multicast: 254
collisions: 0
rx_over_errors: 0
rx_crc_errors: 0
rx_frame_errors: 0
rx_fifo_errors: 0
rx_missed_errors: 0
tx_aborted_errors: 0
tx_carrier_errors: 0
tx_fifo_errors: 0
tx_heartbeat_errors: 0
rx_pkts_nic: 728170
tx_pkts_nic: 8
rx_bytes_nic: 1010157240
tx_bytes_nic: 680
lsc_int: 1
tx_busy: 0
non_eop_descs: 0
broadcast: 63
rx_no_buffer_count: 0
tx_timeout_count: 0
tx_restart_queue: 0
rx_long_length_errors: 0
rx_short_length_errors: 0
tx_flow_control_xon: 0
rx_flow_control_xon: 0
tx_flow_control_xoff: 0
rx_flow_control_xoff: 0
rx_csum_offload_errors: 0
alloc_rx_page_failed: 0
alloc_rx_buff_failed: 0
lro_aggregated: 0
lro_flushed: 0
rx_no_dma_resources: 0
hw_rsc_aggregated: 0
hw_rsc_flushed: 0
fdir_match: 0
fdir_miss: 727925
fdir_overflow: 0
fcoe_bad_fccrc: 0
fcoe_last_errors: 0
rx_fcoe_dropped: 0
rx_fcoe_packets: 0
rx_fcoe_dwords: 0
fcoe_noddp: 0
fcoe_noddp_ext_buff: 0
tx_fcoe_packets: 0
tx_fcoe_dwords: 0
os2bmc_rx_by_bmc: 0
os2bmc_tx_by_bmc: 0
os2bmc_tx_by_host: 0
os2bmc_rx_by_host: 0
tx_hwtstamp_timeouts: 0
rx_hwtstamp_cleared: 0
tx_queue_0_packets: 0
tx_queue_0_bytes: 0
tx_queue_0_bp_napi_yield: 0
tx_queue_0_bp_misses: 0
tx_queue_0_bp_cleaned: 0
tx_queue_1_packets: 0
tx_queue_1_bytes: 0
tx_queue_1_bp_napi_yield: 0
tx_queue_1_bp_misses: 0
tx_queue_1_bp_cleaned: 0
tx_queue_2_packets: 0
tx_queue_2_bytes: 0
tx_queue_2_bp_napi_yield: 0
tx_queue_2_bp_misses: 0
tx_queue_2_bp_cleaned: 0
tx_queue_3_packets: 0
tx_queue_3_bytes: 0
tx_queue_3_bp_napi_yield: 0
tx_queue_3_bp_misses: 0
tx_queue_3_bp_cleaned: 0
tx_queue_4_packets: 0
tx_queue_4_bytes: 0
tx_queue_4_bp_napi_yield: 0
tx_queue_4_bp_misses: 0
tx_queue_4_bp_cleaned: 0
tx_queue_5_packets: 0
tx_queue_5_bytes: 0
tx_queue_5_bp_napi_yield: 0
tx_queue_5_bp_misses: 0
tx_queue_5_bp_cleaned: 0
tx_queue_6_packets: 0
tx_queue_6_bytes: 0
tx_queue_6_bp_napi_yield: 0
tx_queue_6_bp_misses: 0
tx_queue_6_bp_cleaned: 0
tx_queue_7_packets: 0
tx_queue_7_bytes: 0
tx_queue_7_bp_napi_yield: 0
tx_queue_7_bp_misses: 0
tx_queue_7_bp_cleaned: 0
tx_queue_8_packets: 0
tx_queue_8_bytes: 0
tx_queue_8_bp_napi_yield: 0
tx_queue_8_bp_misses: 0
tx_queue_8_bp_cleaned: 0
tx_queue_9_packets: 0
tx_queue_9_bytes: 0
tx_queue_9_bp_napi_yield: 0
tx_queue_9_bp_misses: 0
tx_queue_9_bp_cleaned: 0
tx_queue_10_packets: 0
tx_queue_10_bytes: 0
tx_queue_10_bp_napi_yield: 0
tx_queue_10_bp_misses: 0
tx_queue_10_bp_cleaned: 0
tx_queue_11_packets: 0
tx_queue_11_bytes: 0
tx_queue_11_bp_napi_yield: 0
tx_queue_11_bp_misses: 0
tx_queue_11_bp_cleaned: 0
tx_queue_12_packets: 0
tx_queue_12_bytes: 0
tx_queue_12_bp_napi_yield: 0
tx_queue_12_bp_misses: 0
tx_queue_12_bp_cleaned: 0
tx_queue_13_packets: 0
tx_queue_13_bytes: 0
tx_queue_13_bp_napi_yield: 0
tx_queue_13_bp_misses: 0
tx_queue_13_bp_cleaned: 0
tx_queue_14_packets: 0
tx_queue_14_bytes: 0
tx_queue_14_bp_napi_yield: 0
tx_queue_14_bp_misses: 0
tx_queue_14_bp_cleaned: 0
tx_queue_15_packets: 0
tx_queue_15_bytes: 0
tx_queue_15_bp_napi_yield: 0
tx_queue_15_bp_misses: 0
tx_queue_15_bp_cleaned: 0
tx_queue_16_packets: 0
tx_queue_16_bytes: 0
tx_queue_16_bp_napi_yield: 0
tx_queue_16_bp_misses: 0
tx_queue_16_bp_cleaned: 0
tx_queue_17_packets: 0
tx_queue_17_bytes: 0
tx_queue_17_bp_napi_yield: 0
tx_queue_17_bp_misses: 0
tx_queue_17_bp_cleaned: 0
tx_queue_18_packets: 0
tx_queue_18_bytes: 0
tx_queue_18_bp_napi_yield: 0
tx_queue_18_bp_misses: 0
tx_queue_18_bp_cleaned: 0
tx_queue_19_packets: 0
tx_queue_19_bytes: 0
tx_queue_19_bp_napi_yield: 0
tx_queue_19_bp_misses: 0
tx_queue_19_bp_cleaned: 0
tx_queue_20_packets: 0
tx_queue_20_bytes: 0
tx_queue_20_bp_napi_yield: 0
tx_queue_20_bp_misses: 0
tx_queue_20_bp_cleaned: 0
tx_queue_21_packets: 0
tx_queue_21_bytes: 0
tx_queue_21_bp_napi_yield: 0
tx_queue_21_bp_misses: 0
tx_queue_21_bp_cleaned: 0
tx_queue_22_packets: 0
tx_queue_22_bytes: 0
tx_queue_22_bp_napi_yield: 0
tx_queue_22_bp_misses: 0
tx_queue_22_bp_cleaned: 0
tx_queue_23_packets: 0
tx_queue_23_bytes: 0
tx_queue_23_bp_napi_yield: 0
tx_queue_23_bp_misses: 0
tx_queue_23_bp_cleaned: 0
tx_queue_24_packets: 0
tx_queue_24_bytes: 0
tx_queue_24_bp_napi_yield: 0
tx_queue_24_bp_misses: 0
tx_queue_24_bp_cleaned: 0
tx_queue_25_packets: 0
tx_queue_25_bytes: 0
tx_queue_25_bp_napi_yield: 0
tx_queue_25_bp_misses: 0
tx_queue_25_bp_cleaned: 0
tx_queue_26_packets: 0
tx_queue_26_bytes: 0
tx_queue_26_bp_napi_yield: 0
tx_queue_26_bp_misses: 0
tx_queue_26_bp_cleaned: 0
tx_queue_27_packets: 8
tx_queue_27_bytes: 648
tx_queue_27_bp_napi_yield: 0
tx_queue_27_bp_misses: 0
tx_queue_27_bp_cleaned: 0
tx_queue_28_packets: 0
tx_queue_28_bytes: 0
tx_queue_28_bp_napi_yield: 0
tx_queue_28_bp_misses: 0
tx_queue_28_bp_cleaned: 0
tx_queue_29_packets: 0
tx_queue_29_bytes: 0
tx_queue_29_bp_napi_yield: 0
tx_queue_29_bp_misses: 0
tx_queue_29_bp_cleaned: 0
tx_queue_30_packets: 0
tx_queue_30_bytes: 0
tx_queue_30_bp_napi_yield: 0
tx_queue_30_bp_misses: 0
tx_queue_30_bp_cleaned: 0
tx_queue_31_packets: 0
tx_queue_31_bytes: 0
tx_queue_31_bp_napi_yield: 0
tx_queue_31_bp_misses: 0
tx_queue_31_bp_cleaned: 0
tx_queue_32_packets: 0
tx_queue_32_bytes: 0
tx_queue_32_bp_napi_yield: 0
tx_queue_32_bp_misses: 0
tx_queue_32_bp_cleaned: 0
tx_queue_33_packets: 0
tx_queue_33_bytes: 0
tx_queue_33_bp_napi_yield: 0
tx_queue_33_bp_misses: 0
tx_queue_33_bp_cleaned: 0
tx_queue_34_packets: 0
tx_queue_34_bytes: 0
tx_queue_34_bp_napi_yield: 0
tx_queue_34_bp_misses: 0
tx_queue_34_bp_cleaned: 0
tx_queue_35_packets: 0
tx_queue_35_bytes: 0
tx_queue_35_bp_napi_yield: 0
tx_queue_35_bp_misses: 0
tx_queue_35_bp_cleaned: 0
tx_queue_36_packets: 0
tx_queue_36_bytes: 0
tx_queue_36_bp_napi_yield: 0
tx_queue_36_bp_misses: 0
tx_queue_36_bp_cleaned: 0
tx_queue_37_packets: 0
tx_queue_37_bytes: 0
tx_queue_37_bp_napi_yield: 0
tx_queue_37_bp_misses: 0
tx_queue_37_bp_cleaned: 0
tx_queue_38_packets: 0
tx_queue_38_bytes: 0
tx_queue_38_bp_napi_yield: 0
tx_queue_38_bp_misses: 0
tx_queue_38_bp_cleaned: 0
tx_queue_39_packets: 0
tx_queue_39_bytes: 0
tx_queue_39_bp_napi_yield: 0
tx_queue_39_bp_misses: 0
tx_queue_39_bp_cleaned: 0
tx_queue_40_packets: 0
tx_queue_40_bytes: 0
tx_queue_40_bp_napi_yield: 0
tx_queue_40_bp_misses: 0
tx_queue_40_bp_cleaned: 0
tx_queue_41_packets: 0
tx_queue_41_bytes: 0
tx_queue_41_bp_napi_yield: 0
tx_queue_41_bp_misses: 0
tx_queue_41_bp_cleaned: 0
tx_queue_42_packets: 0
tx_queue_42_bytes: 0
tx_queue_42_bp_napi_yield: 0
tx_queue_42_bp_misses: 0
tx_queue_42_bp_cleaned: 0
tx_queue_43_packets: 0
tx_queue_43_bytes: 0
tx_queue_43_bp_napi_yield: 0
tx_queue_43_bp_misses: 0
tx_queue_43_bp_cleaned: 0
tx_queue_44_packets: 0
tx_queue_44_bytes: 0
tx_queue_44_bp_napi_yield: 0
tx_queue_44_bp_misses: 0
tx_queue_44_bp_cleaned: 0
tx_queue_45_packets: 0
tx_queue_45_bytes: 0
tx_queue_45_bp_napi_yield: 0
tx_queue_45_bp_misses: 0
tx_queue_45_bp_cleaned: 0
tx_queue_46_packets: 0
tx_queue_46_bytes: 0
tx_queue_46_bp_napi_yield: 0
tx_queue_46_bp_misses: 0
tx_queue_46_bp_cleaned: 0
tx_queue_47_packets: 0
tx_queue_47_bytes: 0
tx_queue_47_bp_napi_yield: 0
tx_queue_47_bp_misses: 0
tx_queue_47_bp_cleaned: 0
tx_queue_48_packets: 0
tx_queue_48_bytes: 0
tx_queue_48_bp_napi_yield: 0
tx_queue_48_bp_misses: 0
tx_queue_48_bp_cleaned: 0
tx_queue_49_packets: 0
tx_queue_49_bytes: 0
tx_queue_49_bp_napi_yield: 0
tx_queue_49_bp_misses: 0
tx_queue_49_bp_cleaned: 0
tx_queue_50_packets: 0
tx_queue_50_bytes: 0
tx_queue_50_bp_napi_yield: 0
tx_queue_50_bp_misses: 0
tx_queue_50_bp_cleaned: 0
tx_queue_51_packets: 0
tx_queue_51_bytes: 0
tx_queue_51_bp_napi_yield: 0
tx_queue_51_bp_misses: 0
tx_queue_51_bp_cleaned: 0
tx_queue_52_packets: 0
tx_queue_52_bytes: 0
tx_queue_52_bp_napi_yield: 0
tx_queue_52_bp_misses: 0
tx_queue_52_bp_cleaned: 0
tx_queue_53_packets: 0
tx_queue_53_bytes: 0
tx_queue_53_bp_napi_yield: 0
tx_queue_53_bp_misses: 0
tx_queue_53_bp_cleaned: 0
tx_queue_54_packets: 0
tx_queue_54_bytes: 0
tx_queue_54_bp_napi_yield: 0
tx_queue_54_bp_misses: 0
tx_queue_54_bp_cleaned: 0
tx_queue_55_packets: 0
tx_queue_55_bytes: 0
tx_queue_55_bp_napi_yield: 0
tx_queue_55_bp_misses: 0
tx_queue_55_bp_cleaned: 0
tx_queue_56_packets: 0
tx_queue_56_bytes: 0
tx_queue_56_bp_napi_yield: 0
tx_queue_56_bp_misses: 0
tx_queue_56_bp_cleaned: 0
tx_queue_57_packets: 0
tx_queue_57_bytes: 0
tx_queue_57_bp_napi_yield: 0
tx_queue_57_bp_misses: 0
tx_queue_57_bp_cleaned: 0
tx_queue_58_packets: 0
tx_queue_58_bytes: 0
tx_queue_58_bp_napi_yield: 0
tx_queue_58_bp_misses: 0
tx_queue_58_bp_cleaned: 0
tx_queue_59_packets: 0
tx_queue_59_bytes: 0
tx_queue_59_bp_napi_yield: 0
tx_queue_59_bp_misses: 0
tx_queue_59_bp_cleaned: 0
tx_queue_60_packets: 0
tx_queue_60_bytes: 0
tx_queue_60_bp_napi_yield: 0
tx_queue_60_bp_misses: 0
tx_queue_60_bp_cleaned: 0
tx_queue_61_packets: 0
tx_queue_61_bytes: 0
tx_queue_61_bp_napi_yield: 0
tx_queue_61_bp_misses: 0
tx_queue_61_bp_cleaned: 0
tx_queue_62_packets: 0
tx_queue_62_bytes: 0
tx_queue_62_bp_napi_yield: 0
tx_queue_62_bp_misses: 0
tx_queue_62_bp_cleaned: 0
tx_queue_63_packets: 0
tx_queue_63_bytes: 0
tx_queue_63_bp_napi_yield: 0
tx_queue_63_bp_misses: 0
tx_queue_63_bp_cleaned: 0
tx_queue_64_packets: 0
tx_queue_64_bytes: 0
tx_queue_64_bp_napi_yield: 0
tx_queue_64_bp_misses: 0
tx_queue_64_bp_cleaned: 0
tx_queue_65_packets: 0
tx_queue_65_bytes: 0
tx_queue_65_bp_napi_yield: 0
tx_queue_65_bp_misses: 0
tx_queue_65_bp_cleaned: 0
tx_queue_66_packets: 0
tx_queue_66_bytes: 0
tx_queue_66_bp_napi_yield: 0
tx_queue_66_bp_misses: 0
tx_queue_66_bp_cleaned: 0
tx_queue_67_packets: 0
tx_queue_67_bytes: 0
tx_queue_67_bp_napi_yield: 0
tx_queue_67_bp_misses: 0
tx_queue_67_bp_cleaned: 0
tx_queue_68_packets: 0
tx_queue_68_bytes: 0
tx_queue_68_bp_napi_yield: 0
tx_queue_68_bp_misses: 0
tx_queue_68_bp_cleaned: 0
tx_queue_69_packets: 0
tx_queue_69_bytes: 0
tx_queue_69_bp_napi_yield: 0
tx_queue_69_bp_misses: 0
tx_queue_69_bp_cleaned: 0
tx_queue_70_packets: 0
tx_queue_70_bytes: 0
tx_queue_70_bp_napi_yield: 0
tx_queue_70_bp_misses: 0
tx_queue_70_bp_cleaned: 0
rx_queue_0_packets: 9482
rx_queue_0_bytes: 11643129
rx_queue_0_bp_poll_yield: 0
rx_queue_0_bp_misses: 0
rx_queue_0_bp_cleaned: 0
rx_queue_1_packets: 165724
rx_queue_1_bytes: 232486518
rx_queue_1_bp_poll_yield: 0
rx_queue_1_bp_misses: 0
rx_queue_1_bp_cleaned: 0
rx_queue_2_packets: 80657
rx_queue_2_bytes: 112241119
rx_queue_2_bp_poll_yield: 0
rx_queue_2_bp_misses: 0
rx_queue_2_bp_cleaned: 0
rx_queue_3_packets: 46645
rx_queue_3_bytes: 64303361
rx_queue_3_bp_poll_yield: 0
rx_queue_3_bp_misses: 0
rx_queue_3_bp_cleaned: 0
rx_queue_4_packets: 39907
rx_queue_4_bytes: 55708359
rx_queue_4_bp_poll_yield: 0
rx_queue_4_bp_misses: 0
rx_queue_4_bp_cleaned: 0
rx_queue_5_packets: 57077
rx_queue_5_bytes: 77940273
rx_queue_5_bp_poll_yield: 0
rx_queue_5_bp_misses: 0
rx_queue_5_bp_cleaned: 0
rx_queue_6_packets: 12543
rx_queue_6_bytes: 17162117
rx_queue_6_bp_poll_yield: 0
rx_queue_6_bp_misses: 0
rx_queue_6_bp_cleaned: 0
rx_queue_7_packets: 45791
rx_queue_7_bytes: 63757988
rx_queue_7_bp_poll_yield: 0
rx_queue_7_bp_misses: 0
rx_queue_7_bp_cleaned: 0
rx_queue_8_packets: 10118
rx_queue_8_bytes: 13687646
rx_queue_8_bp_poll_yield: 0
rx_queue_8_bp_misses: 0
rx_queue_8_bp_cleaned: 0
rx_queue_9_packets: 69990
rx_queue_9_bytes: 97934717
rx_queue_9_bp_poll_yield: 0
rx_queue_9_bp_misses: 0
rx_queue_9_bp_cleaned: 0
rx_queue_10_packets: 5588
rx_queue_10_bytes: 7125456
rx_queue_10_bp_poll_yield: 0
rx_queue_10_bp_misses: 0
rx_queue_10_bp_cleaned: 0
rx_queue_11_packets: 73009
rx_queue_11_bytes: 101591101
rx_queue_11_bp_poll_yield: 0
rx_queue_11_bp_misses: 0
rx_queue_11_bp_cleaned: 0
rx_queue_12_packets: 6666
rx_queue_12_bytes: 8709930
rx_queue_12_bp_poll_yield: 0
rx_queue_12_bp_misses: 0
rx_queue_12_bp_cleaned: 0
rx_queue_13_packets: 86238
rx_queue_13_bytes: 119855689
rx_queue_13_bp_poll_yield: 0
rx_queue_13_bp_misses: 0
rx_queue_13_bp_cleaned: 0
rx_queue_14_packets: 10094
rx_queue_14_bytes: 12339328
rx_queue_14_bp_poll_yield: 0
rx_queue_14_bp_misses: 0
rx_queue_14_bp_cleaned: 0
rx_queue_15_packets: 8641
rx_queue_15_bytes: 10757829
rx_queue_15_bp_poll_yield: 0
rx_queue_15_bp_misses: 0
rx_queue_15_bp_cleaned: 0
rx_queue_16_packets: 0
rx_queue_16_bytes: 0
rx_queue_16_bp_poll_yield: 0
rx_queue_16_bp_misses: 0
rx_queue_16_bp_cleaned: 0
rx_queue_17_packets: 0
rx_queue_17_bytes: 0
rx_queue_17_bp_poll_yield: 0
rx_queue_17_bp_misses: 0
rx_queue_17_bp_cleaned: 0
rx_queue_18_packets: 0
rx_queue_18_bytes: 0
rx_queue_18_bp_poll_yield: 0
rx_queue_18_bp_misses: 0
rx_queue_18_bp_cleaned: 0
rx_queue_19_packets: 0
rx_queue_19_bytes: 0
rx_queue_19_bp_poll_yield: 0
rx_queue_19_bp_misses: 0
rx_queue_19_bp_cleaned: 0
rx_queue_20_packets: 0
rx_queue_20_bytes: 0
rx_queue_20_bp_poll_yield: 0
rx_queue_20_bp_misses: 0
rx_queue_20_bp_cleaned: 0
rx_queue_21_packets: 0
rx_queue_21_bytes: 0
rx_queue_21_bp_poll_yield: 0
rx_queue_21_bp_misses: 0
rx_queue_21_bp_cleaned: 0
rx_queue_22_packets: 0
rx_queue_22_bytes: 0
rx_queue_22_bp_poll_yield: 0
rx_queue_22_bp_misses: 0
rx_queue_22_bp_cleaned: 0
rx_queue_23_packets: 0
rx_queue_23_bytes: 0
rx_queue_23_bp_poll_yield: 0
rx_queue_23_bp_misses: 0
rx_queue_23_bp_cleaned: 0
rx_queue_24_packets: 0
rx_queue_24_bytes: 0
rx_queue_24_bp_poll_yield: 0
rx_queue_24_bp_misses: 0
rx_queue_24_bp_cleaned: 0
rx_queue_25_packets: 0
rx_queue_25_bytes: 0
rx_queue_25_bp_poll_yield: 0
rx_queue_25_bp_misses: 0
rx_queue_25_bp_cleaned: 0
rx_queue_26_packets: 0
rx_queue_26_bytes: 0
rx_queue_26_bp_poll_yield: 0
rx_queue_26_bp_misses: 0
rx_queue_26_bp_cleaned: 0
rx_queue_27_packets: 0
rx_queue_27_bytes: 0
rx_queue_27_bp_poll_yield: 0
rx_queue_27_bp_misses: 0
rx_queue_27_bp_cleaned: 0
rx_queue_28_packets: 0
rx_queue_28_bytes: 0
rx_queue_28_bp_poll_yield: 0
rx_queue_28_bp_misses: 0
rx_queue_28_bp_cleaned: 0
rx_queue_29_packets: 0
rx_queue_29_bytes: 0
rx_queue_29_bp_poll_yield: 0
rx_queue_29_bp_misses: 0
rx_queue_29_bp_cleaned: 0
rx_queue_30_packets: 0
rx_queue_30_bytes: 0
rx_queue_30_bp_poll_yield: 0
rx_queue_30_bp_misses: 0
rx_queue_30_bp_cleaned: 0
rx_queue_31_packets: 0
rx_queue_31_bytes: 0
rx_queue_31_bp_poll_yield: 0
rx_queue_31_bp_misses: 0
rx_queue_31_bp_cleaned: 0
rx_queue_32_packets: 0
rx_queue_32_bytes: 0
rx_queue_32_bp_poll_yield: 0
rx_queue_32_bp_misses: 0
rx_queue_32_bp_cleaned: 0
rx_queue_33_packets: 0
rx_queue_33_bytes: 0
rx_queue_33_bp_poll_yield: 0
rx_queue_33_bp_misses: 0
rx_queue_33_bp_cleaned: 0
rx_queue_34_packets: 0
rx_queue_34_bytes: 0
rx_queue_34_bp_poll_yield: 0
rx_queue_34_bp_misses: 0
rx_queue_34_bp_cleaned: 0
rx_queue_35_packets: 0
rx_queue_35_bytes: 0
rx_queue_35_bp_poll_yield: 0
rx_queue_35_bp_misses: 0
rx_queue_35_bp_cleaned: 0
rx_queue_36_packets: 0
rx_queue_36_bytes: 0
rx_queue_36_bp_poll_yield: 0
rx_queue_36_bp_misses: 0
rx_queue_36_bp_cleaned: 0
rx_queue_37_packets: 0
rx_queue_37_bytes: 0
rx_queue_37_bp_poll_yield: 0
rx_queue_37_bp_misses: 0
rx_queue_37_bp_cleaned: 0
rx_queue_38_packets: 0
rx_queue_38_bytes: 0
rx_queue_38_bp_poll_yield: 0
rx_queue_38_bp_misses: 0
rx_queue_38_bp_cleaned: 0
rx_queue_39_packets: 0
rx_queue_39_bytes: 0
rx_queue_39_bp_poll_yield: 0
rx_queue_39_bp_misses: 0
rx_queue_39_bp_cleaned: 0
rx_queue_40_packets: 0
rx_queue_40_bytes: 0
rx_queue_40_bp_poll_yield: 0
rx_queue_40_bp_misses: 0
rx_queue_40_bp_cleaned: 0
rx_queue_41_packets: 0
rx_queue_41_bytes: 0
rx_queue_41_bp_poll_yield: 0
rx_queue_41_bp_misses: 0
rx_queue_41_bp_cleaned: 0
rx_queue_42_packets: 0
rx_queue_42_bytes: 0
rx_queue_42_bp_poll_yield: 0
rx_queue_42_bp_misses: 0
rx_queue_42_bp_cleaned: 0
rx_queue_43_packets: 0
rx_queue_43_bytes: 0
rx_queue_43_bp_poll_yield: 0
rx_queue_43_bp_misses: 0
rx_queue_43_bp_cleaned: 0
rx_queue_44_packets: 0
rx_queue_44_bytes: 0
rx_queue_44_bp_poll_yield: 0
rx_queue_44_bp_misses: 0
rx_queue_44_bp_cleaned: 0
rx_queue_45_packets: 0
rx_queue_45_bytes: 0
rx_queue_45_bp_poll_yield: 0
rx_queue_45_bp_misses: 0
rx_queue_45_bp_cleaned: 0
rx_queue_46_packets: 0
rx_queue_46_bytes: 0
rx_queue_46_bp_poll_yield: 0
rx_queue_46_bp_misses: 0
rx_queue_46_bp_cleaned: 0
rx_queue_47_packets: 0
rx_queue_47_bytes: 0
rx_queue_47_bp_poll_yield: 0
rx_queue_47_bp_misses: 0
rx_queue_47_bp_cleaned: 0
rx_queue_48_packets: 0
rx_queue_48_bytes: 0
rx_queue_48_bp_poll_yield: 0
rx_queue_48_bp_misses: 0
rx_queue_48_bp_cleaned: 0
rx_queue_49_packets: 0
rx_queue_49_bytes: 0
rx_queue_49_bp_poll_yield: 0
rx_queue_49_bp_misses: 0
rx_queue_49_bp_cleaned: 0
rx_queue_50_packets: 0
rx_queue_50_bytes: 0
rx_queue_50_bp_poll_yield: 0
rx_queue_50_bp_misses: 0
rx_queue_50_bp_cleaned: 0
rx_queue_51_packets: 0
rx_queue_51_bytes: 0
rx_queue_51_bp_poll_yield: 0
rx_queue_51_bp_misses: 0
rx_queue_51_bp_cleaned: 0
rx_queue_52_packets: 0
rx_queue_52_bytes: 0
rx_queue_52_bp_poll_yield: 0
rx_queue_52_bp_misses: 0
rx_queue_52_bp_cleaned: 0
rx_queue_53_packets: 0
rx_queue_53_bytes: 0
rx_queue_53_bp_poll_yield: 0
rx_queue_53_bp_misses: 0
rx_queue_53_bp_cleaned: 0
rx_queue_54_packets: 0
rx_queue_54_bytes: 0
rx_queue_54_bp_poll_yield: 0
rx_queue_54_bp_misses: 0
rx_queue_54_bp_cleaned: 0
rx_queue_55_packets: 0
rx_queue_55_bytes: 0
rx_queue_55_bp_poll_yield: 0
rx_queue_55_bp_misses: 0
rx_queue_55_bp_cleaned: 0
rx_queue_56_packets: 0
rx_queue_56_bytes: 0
rx_queue_56_bp_poll_yield: 0
rx_queue_56_bp_misses: 0
rx_queue_56_bp_cleaned: 0
rx_queue_57_packets: 0
rx_queue_57_bytes: 0
rx_queue_57_bp_poll_yield: 0
rx_queue_57_bp_misses: 0
rx_queue_57_bp_cleaned: 0
rx_queue_58_packets: 0
rx_queue_58_bytes: 0
rx_queue_58_bp_poll_yield: 0
rx_queue_58_bp_misses: 0
rx_queue_58_bp_cleaned: 0
rx_queue_59_packets: 0
rx_queue_59_bytes: 0
rx_queue_59_bp_poll_yield: 0
rx_queue_59_bp_misses: 0
rx_queue_59_bp_cleaned: 0
rx_queue_60_packets: 0
rx_queue_60_bytes: 0
rx_queue_60_bp_poll_yield: 0
rx_queue_60_bp_misses: 0
rx_queue_60_bp_cleaned: 0
rx_queue_61_packets: 0
rx_queue_61_bytes: 0
rx_queue_61_bp_poll_yield: 0
rx_queue_61_bp_misses: 0
rx_queue_61_bp_cleaned: 0
rx_queue_62_packets: 0
rx_queue_62_bytes: 0
rx_queue_62_bp_poll_yield: 0
rx_queue_62_bp_misses: 0
rx_queue_62_bp_cleaned: 0
rx_queue_63_packets: 0
rx_queue_63_bytes: 0
rx_queue_63_bp_poll_yield: 0
rx_queue_63_bp_misses: 0
rx_queue_63_bp_cleaned: 0
rx_queue_64_packets: 0
rx_queue_64_bytes: 0
rx_queue_64_bp_poll_yield: 0
rx_queue_64_bp_misses: 0
rx_queue_64_bp_cleaned: 0
rx_queue_65_packets: 0
rx_queue_65_bytes: 0
rx_queue_65_bp_poll_yield: 0
rx_queue_65_bp_misses: 0
rx_queue_65_bp_cleaned: 0
rx_queue_66_packets: 0
rx_queue_66_bytes: 0
rx_queue_66_bp_poll_yield: 0
rx_queue_66_bp_misses: 0
rx_queue_66_bp_cleaned: 0
rx_queue_67_packets: 0
rx_queue_67_bytes: 0
rx_queue_67_bp_poll_yield: 0
rx_queue_67_bp_misses: 0
rx_queue_67_bp_cleaned: 0
rx_queue_68_packets: 0
rx_queue_68_bytes: 0
rx_queue_68_bp_poll_yield: 0
rx_queue_68_bp_misses: 0
rx_queue_68_bp_cleaned: 0
rx_queue_69_packets: 0
rx_queue_69_bytes: 0
rx_queue_69_bp_poll_yield: 0
rx_queue_69_bp_misses: 0
rx_queue_69_bp_cleaned: 0
rx_queue_70_packets: 0
rx_queue_70_bytes: 0
rx_queue_70_bp_poll_yield: 0
rx_queue_70_bp_misses: 0
rx_queue_70_bp_cleaned: 0
tx_pb_0_pxon: 0
tx_pb_0_pxoff: 0
tx_pb_1_pxon: 0
tx_pb_1_pxoff: 0
tx_pb_2_pxon: 0
tx_pb_2_pxoff: 0
tx_pb_3_pxon: 0
tx_pb_3_pxoff: 0
tx_pb_4_pxon: 0
tx_pb_4_pxoff: 0
tx_pb_5_pxon: 0
tx_pb_5_pxoff: 0
tx_pb_6_pxon: 0
tx_pb_6_pxoff: 0
tx_pb_7_pxon: 0
tx_pb_7_pxoff: 0
rx_pb_0_pxon: 0
rx_pb_0_pxoff: 0
rx_pb_1_pxon: 0
rx_pb_1_pxoff: 0
rx_pb_2_pxon: 0
rx_pb_2_pxoff: 0
rx_pb_3_pxon: 0
rx_pb_3_pxoff: 0
rx_pb_4_pxon: 0
rx_pb_4_pxoff: 0
rx_pb_5_pxon: 0
rx_pb_5_pxoff: 0
rx_pb_6_pxon: 0
rx_pb_6_pxoff: 0
rx_pb_7_pxon: 0
rx_pb_7_pxoff: 0