I'm posting here in the hope that someone has seen symptoms like this and can
offer some insight as to what's going on.
Executive summary: We have a newly-installed E1G44ET2BLK 4-port server adapter;
following a server reboot, we found that the ethernet jacks on that card had
been reordered! How did that happen, and how do we prevent it?
Here's how ifconfig saw this card after it was installed, with an ethernet
cable plugged into the bottom jack:
igb0: flags=1000802<BROADCAST,MULTICAST,IPv4> mtu 1500 index 13
inet 0.0.0.0 netmask 0
igb1: flags=1000802<BROADCAST,MULTICAST,IPv4> mtu 1500 index 14
inet 0.0.0.0 netmask 0
igb2: flags=1000802<BROADCAST,MULTICAST,IPv4> mtu 1500 index 12
inet 0.0.0.0 netmask 0
igb3: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU> mtu 1500 index 5
inet 171.64.175.54 netmask fffff800 broadcast 171.64.175.255
We checked every port on the card one at a time by plugging in an ethernet
cable and running ifconfig to see which network interface showed valid link
status (the RUNNING bit). We labeled the jacks b0 through b3 on the back panel; no
surprise, they were ordered top-to-bottom, b0 to b3.
After a server reboot, ifconfig showed the following:
igb0: flags=1000802<BROADCAST,MULTICAST,IPv4> mtu 1500 index 13
inet 0.0.0.0 netmask 0
igb1: flags=1000802<BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 14
inet 0.0.0.0 netmask 0
igb2: flags=1000802<BROADCAST,MULTICAST,IPv4> mtu 1500 index 12
inet 0.0.0.0 netmask 0
igb3: flags=1001000843<UP,BROADCAST,MULTICAST,IPv4,FIXEDMTU> mtu 1500 index 5
inet 171.64.175.54 netmask fffff800 broadcast 171.64.175.255
Notice that igb3 is no longer RUNNING; igb1 shows as RUNNING, instead. Yet the
cable had not been moved! It's still plugged in to the bottom jack. What's
changed is the mapping from network device name to physical port: the bottom
jack, which used to be igb3, is now igb1. And igb1, which used to be the 2nd
jack down from the top, is now the bottom jack. Our carefully-applied labels
are now wrong!
To restore network connectivity, I had to move the cable from the bottom jack
to the 2nd jack down. This restored RUNNING status to igb3.
We can't live with network jacks that move around on our server every time
we reboot. How is this remapping happening? Is this a feature of the E1G44ET2
card? An issue with the card's firmware? A device driver or OS problem?
The OS is NexentaStor v3.1.3 Enterprise (SunOS Release 5.11, OpenSolaris
Build 134). Server node is based on the Xyratex OneStorHS 1235T chassis.
Driver version is apparently 1.1.18:
# modinfo | grep igb
179 fffffffff8084000 1e178 284 1 igb (Intel 1Gb Ethernet)
# strings /kernel/drv/amd64/igb | grep igb
igb 1.1.18
-- Kai Lanz Stanford University School of Earth Sciences