I have configured host with two PF and 16 x2 VF in debian system. It worked well if I use the PF/VF as standalone NIC, but not functional with OVS bridge.
The testing scenario as below:
eth0 (PF): 10.250.11.103/24
eth11 (VF 9): 10.250.11.113/24
Bridge "vmbr1"
Port "eth10"
Interface "eth10" (VF 8)
Port "vmbr1"
Interface "vmbr1"
type: internal
Port "in1"
Interface "in1"
type: internal
ovs_version: "2.6.0"
in1: 10.250.11.34/24
I try the following icmp test:
ping 10.250.11.254 (reply good)
ping 10.250.11.254 -I eth11 (reply good)
ping 10.250.11.254 -I in1 (NO response)
ip link show my VF as:
-------------------------------
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 0c:c4:7a:df:36:c6 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 6e:74:9e:65:58:eb, spoof checking on, link-state auto
vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
vf 2 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
vf 3 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
vf 4 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
vf 5 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
vf 6 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
vf 7 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
vf 8 MAC da:c7:06:0b:5b:1f, spoof checking on, link-state auto
vf 9 MAC ea:97:12:4e:b4:6b, spoof checking on, link-state auto
vf 10 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
vf 11 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
vf 12 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
vf 13 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
vf 14 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
vf 15 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
arp table shown as:
-------------------------------
Address HWtype HWaddress Flags Mask Iface
10.250.11.254 ether 6c:3b:6b:e9:7c:e1 C eth11
10.250.11.254 ether 6c:3b:6b:e9:7c:e1 C eth0
10.250.11.254 ether 6c:3b:6b:e9:7c:e1 C in1
But arp is not stable, it disappear very quickly:
------------------------------------------------------------
Address HWtype HWaddress Flags Mask Iface
10.250.11.254 (incomplete) in1
10.250.11.254 ether 6c:3b:6b:e9:7c:e1 C eth0
10.250.11.254 ether 6c:3b:6b:e9:7c:e1 C eth11
The strange thing is:
ping via in1 will not fill up the arp table for in1 Iface (always show incomplete), but ping via other two iface seems fill the mac address into in1 some time.
I have tcpdump to in1 and found it send out ARP request broadcast, but did not get any ARP reply from host 254.
ovs-appctl fdb/show vmbr1:
-----------------------------------------
port VLAN MAC Age
2 0 72:fc:b4:3a:e6:0f 78
1 0 6c:3b:6b:e9:7c:e1 38
1 0 6c:3b:6b:f5:96:e6 3
1 0 88:43:e1:dd:9c:80 1
This looks persist and will not change whatever I ping via any Iface.
I have also try to change VF port, but seems none of them work with OVS bridge.
But the bridge worked if assigned the port to eth0 or eth1 PF.
uname -a:
Linux pve3 4.4.62-1-pve #1 SMP PVE 4.4.62-88 (Thu, 18 May 2017 09:18:43 +0200) x86_64 GNU/Linux
/etc/network/interfaces:
----------------------------------
auto eth0
iface eth0 inet static
address 10.250.11.103
netmask 255.255.255.0
gateway 10.250.11.254
auto eth11
iface eth11 inet static
address 10.250.11.113
netmask 255.255.255.0
allow-vmbr1 eth10
iface eth10 inet manual
ovs_type OVSPort
ovs_bridge vmbr1
auto vmbr1
iface vmbr1 inet manual
ovs_type OVSBridge
ovs_ports eth10 in1
allow-vmbr1 in1
iface in1 inet static
address 10.250.11.34
netmask 255.255.255.0
ovs_type OVSIntPort
ovs_bridge vmbr1
Here is ethtool info for eth10:
-------------------------------
# ethtool -i eth10
driver: ixgbevf
version: 2.12.1-k
firmware-version:
bus-info: 0000:05:12.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no
#ethtool -k eth10
Features for eth10:
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: on
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: on
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp6-segmentation: on
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on [fixed]
tx-vlan-offload: on [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on [fixed]
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
busy-poll: on [fixed]
hw-tc-offload: off [fixed]
-------------------------------
Is there any advice?