Symptoms
- vmnic enumeration changed due to change in PCI Slot id.
- Duplicate mac address noticed for two vmnics in esx.conf file.
Cause
The vmnic label changes due to change in PCI Slot ID and many other factors.
Resolution
- Open the /etc/vmware/esx.conf file and check if there are duplicate mac address for multiple nics.
vmnic10 - vmnic27 - 00:25:b5:00:00:2c
vmnic11 - vmnic28 - 00:25:b5:00:00:5c
vmnic12 - vmnic29 - 00:25:b5:00:00:4c
vmnic13 - vmnic30 - 00:25:b5:00:00:7c
vmnic14 - vmnic31 - 00:25:b5:00:00:6c
vmnic16 - vmnic33 - 00:25:b5:00:00:8c
vmnic15 - vmnic32 - 00:25:b5:00:00:9c
vmnic17 - vmnic34 - 00:25:b5:00:00:ad
- So here vmnic10 label has changed to vmnic27 after the server migration activity since both the labels are pointing to the same physical mac address 00:25:b5:00:00:2c.
- Open the lspci.txt located under commands directory in scripts folder and check for the current mapping of vmnic to mac address.
000:005:00.0 Network controller: Cisco Systems Inc Cisco VIC Ethernet NIC [vmnic0]
000:006:00.0 Network controller: Cisco Systems Inc Cisco VIC Ethernet NIC [vmnic8]
000:007:00.0 Network controller: Cisco Systems Inc Cisco VIC Ethernet NIC [vmnic16]
000:008:00.0 Network controller: Cisco Systems Inc Cisco VIC Ethernet NIC [vmnic20]
000:009:00.0 Network controller: Cisco Systems Inc Cisco VIC Ethernet NIC [vmnic21]
000:010:00.0 Network controller: Cisco Systems Inc Cisco VIC Ethernet NIC [vmnic22]
000:011:00.0 Network controller: Cisco Systems Inc Cisco VIC Ethernet NIC [vmnic23]
000:012:00.0 Network controller: Cisco Systems Inc Cisco VIC Ethernet NIC [vmnic24]
000:013:00.0 Network controller: Cisco Systems Inc Cisco VIC Ethernet NIC [vmnic25]
000:014:00.0 Network controller: Cisco Systems Inc Cisco VIC Ethernet NIC [vmnic26]
000:015:00.0 Network controller: Cisco Systems Inc Cisco VIC Ethernet NIC [vmnic27]
000:016:00.0 Network controller: Cisco Systems Inc Cisco VIC Ethernet NIC [vmnic28]
000:017:00.0 Network controller: Cisco Systems Inc Cisco VIC Ethernet NIC [vmnic29]
000:018:00.0 Network controller: Cisco Systems Inc Cisco VIC Ethernet NIC [vmnic30]
000:019:00.0 Network controller: Cisco Systems Inc Cisco VIC Ethernet NIC [vmnic31]
000:020:00.0 Network controller: Cisco Systems Inc Cisco VIC Ethernet NIC [vmnic32]
000:021:00.0 Network controller: Cisco Systems Inc Cisco VIC Ethernet NIC [vmnic33]
000:022:00.0 Network controller: Cisco Systems Inc Cisco VIC Ethernet NIC [vmnic34]
000:023:00.0 Network controller: Cisco Systems Inc Cisco VIC Ethernet NIC [vmnic35]
000:024:00.0 Network controller: Cisco Systems Inc Cisco VIC Ethernet NIC [vmnic36]
- Here vmnic27 is mapped to the concerned mac address after ESXi booted up. So the vmnic10 label actually does not exist.
- Goto var/run/log and grep for vmnic10 (old nic label) in old vmkernel logs before the migration happened.
cat /var/run/log/vmkernel.7 | grep -i vmnic10
2013-04-13T12:54:31.211Z cpu22:4771)PCI: 3938: 000:006:00.2 named 'vmnic10' (was '')
2013-04-13T12:54:35.995Z cpu22:4771)VMK_PCI: 684: Device 000:006:00.2 name: vmnic10
2013-04-13T12:54:35.995Z cpu22:4771)Uplink: 5262: Device vmnic10 not yet opened
2013-04-13T12:54:36.057Z cpu10:4771)Uplink: 6912: Opening device vmnic10
2013-04-13T12:54:36.057Z cpu23:4155)DMA: 524: DMA Engine 'vmnic10' created.
2013-04-13T12:54:36.057Z cpu23:4155)IDT: 991: 0xaa< vmnic10-rx-0> exclusive (entropy source), flags 0x10
2013-04-13T12:54:36.057Z cpu23:4155)IDT: 991: 0xb2< vmnic10-tx-0> exclusive (entropy source), flags 0x10
2013-04-13T12:54:36.057Z cpu23:4155)IDT: 991: 0xba< vmnic10-err> exclusive (entropy source), flags 0x10
2013-04-13T12:54:36.057Z cpu23:4155)IDT: 991: 0xc2< vmnic10-notify> exclusive (entropy source), flags 0x10
2013-04-13T12:54:36.913Z cpu22:4118)<6>enic 0000:06:00.2: vmnic10: Link UP
- The older entry of vmkernel logs indicate vmnic10 was slotted on PCI ID: 000:006:00.2.
- grep for vmnic27 (new label) in latest vmkernel logs after the migration completed.
cat var/log/vmkernel.0 | grep -i vmnic27
2013-04-28T02:44:47.299Z cpu17:4771)PCI: 3938: 000:015:00.0 named 'vmnic27' (was '')
2013-04-28T02:44:55.282Z cpu17:4771)VMK_PCI: 684: Device 000:015:00.0 name: vmnic27
2013-04-28T02:44:55.282Z cpu17:4771)Uplink: 5262: Device vmnic27 not yet opened
2013-04-28T02:44:55.396Z cpu21:4771)Uplink: 6912: Opening device vmnic27
2013-04-28T02:44:55.396Z cpu19:4155)DMA: 524: DMA Engine 'vmnic27' created.
2013-04-28T02:44:55.396Z cpu19:4155)IDT: 991: 0xaa< vmnic27-rx-0> exclusive (entropy source), flags 0x10
2013-04-28T02:44:55.396Z cpu19:4155)IDT: 991: 0xb2< vmnic27-tx-0> exclusive (entropy source), flags 0x10
2013-04-28T02:44:55.396Z cpu19:4155)IDT: 991: 0xba< vmnic27-err> exclusive (entropy source), flags 0x10
2013-04-28T02:44:55.396Z cpu19:4155)IDT: 991: 0xc2< vmnic27-notify> exclusive (entropy source), flags 0x10
2013-04-28T02:44:56.446Z cpu17:4113)<6>enic 0000:0f:00.0: vmnic27: Link UP
- The vmkernel logs indicate vmnic27 is slotted on PCI ID: 000:015:00.0.
- Based on the above log snippets, we can confirm that the PCI ID of physical nic with MAC address 00:25:b5:00:00:2c was changed from 000:006:00.2 to 000:015:00.0.
- That resulted in vmnic label to change from 10 to 27.