Greetings all,
We are using Ubunti-Precise 3.2.0-29-generic #46, with KVM version qemu-kvm-1.0 and libvirt 0.9.8. We are using Intel NIC 82599 with ixgbe driver 3.11.33 and in-tree ixgbevf driver 2.2.0-k. We are attaching VFs to KVM instances.
In lspci we see that MSI-X table for each VF has three entries, like this:
03:17.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
Subsystem: Intel Corporation Device 7a11
Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Region 0: [virtual] Memory at df478000 (64-bit, non-prefetchable) [size=16K]
Region 3: [virtual] Memory at df578000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [70] MSI-X: Enable+ Count=3 Masked-
But when attaching the VF to KVM, we see that most of the VFs receive only two IRQs, while only a few receive three IRQs.
For example:
Dec 13 11:04:17 ubuntu-sata-31 kernel: [ 4.448999] ixgbevf 0000:03:17.3: irq 359 for MSI/MSI-X
Dec 13 11:04:17 ubuntu-sata-31 kernel: [ 4.449016] ixgbevf 0000:03:17.3: irq 360 for MSI/MSI-X
Dec 13 11:04:17 ubuntu-sata-31 kernel: [ 4.449033] ixgbevf 0000:03:17.3: irq 361 for MSI/MSI-X
Dec 15 10:23:32 ubuntu-sata-31 kernel: [170107.490908] pci-stub 0000:03:12.2: irq 138 for MSI/MSI-X
Dec 15 10:23:32 ubuntu-sata-31 kernel: [170107.490924] pci-stub 0000:03:12.2: irq 139 for MSI/MSI-X
Dec 15 10:23:32 ubuntu-sata-31 kernel: [170107.730559] pci-stub 0000:03:1c.1: irq 335 for MSI/MSI-X
Dec 15 10:23:32 ubuntu-sata-31 kernel: [170107.730580] pci-stub 0000:03:1c.1: irq 336 for MSI/MSI-X
Initial debugging shows, that KVM reads the PCI configuration space to determine the number of IRQs to assign:
static int assigned_dev_update_msix_mmio(PCIDevice *pci_dev)
{
AssignedDevice *adev = DO_UPCAST(AssignedDevice, dev, pci_dev);
uint16_t entries_nr = 0, entries_max_nr;
int pos = 0, i, r = 0;
uint32_t msg_addr, msg_upper_addr, msg_data, msg_ctrl;
struct kvm_assigned_msix_nr msix_nr;
struct kvm_assigned_msix_entry msix_entry;
void *va = adev->msix_table_page;
pos = pci_find_capability(pci_dev, PCI_CAP_ID_MSIX);
entries_max_nr = *(uint16_t *)(pci_dev->config + pos + 2);
entries_max_nr &= PCI_MSIX_FLAGS_QSIZE;
entries_max_nr += 1;
This yields entries_max_nr=3, as expected.
But then:
/* Get the usable entry number for allocating */
for (i = 0; i < entries_max_nr; i++) {
memcpy(&msg_ctrl, va + i * 16 + 12, 4);
memcpy(&msg_data, va + i * 16 + 8, 4);
/* Ignore unused entry even it's unmasked */
if (msg_data == 0)
continue;
entries_nr ++;
}
And adding some prints shows that the third entry yields msg_data==0, so it is skipped.
ALEXL: PCIDEV(pci-assign) entries_max_nr=3
ALEXL: PCIDEV(pci-assign) entry #0 msg_data=16817
ALEXL: PCIDEV(pci-assign) entry #1 msg_data=16833
ALEXL: PCIDEV(pci-assign) entry #2 msg_data=0
ALEXL: PCIDEV(pci-assign) entries_max_nr=3
ALEXL: PCIDEV(pci-assign) entry #0 msg_data=16849
ALEXL: PCIDEV(pci-assign) entry #1 msg_data=16865
ALEXL: PCIDEV(pci-assign) entry #2 msg_data=0
ALEXL: PCIDEV(pci-assign) entries_max_nr=3
ALEXL: PCIDEV(pci-assign) entry #0 msg_data=16674
ALEXL: PCIDEV(pci-assign) entry #1 msg_data=16706
ALEXL: PCIDEV(pci-assign) entry #2 msg_data=0
We don't know what is the implication of having only two IRQs assigned instead of three. From overall perspective, those KVM instances function normally. Can anybody please comment on this?
When using SR-IOV cards from another vendor, we see that those other cards have 4 entries for each VF, and all those entries have IRQs attached. So we see this issue only with Intel cards.
Thanks,
Alex.