A colleague brought up a very important issue in regards to vPC survivability. There are Nexus vPC Peer-Link interface options. Take a look at the below diagrams. As you can see, we have a big problem here with survivability that can often be overlooked: if the one single vPC layer-3 peer-link interface goes down for whatever reason (cable cut, SFP breaks, module goes down, etc) then the Nexus switches do something extremely horrible. They split-brain-split-forward!
Basically, in the vPC world, all links are set to forward, and nothing blocks at layer-2 because Spanning Tree is not really in place here. Cisco calls it a “split-brain” scenario. If that vPC peer-link fails for any reason, both switches assume the role of primary. This then creates the forwarding-loop-from-hell. Cisco explains it much better here:
If the vPC keepalive link fails first and then a peer link fails, the vPC secondary switch assumes the primary switch role and keeps its vPC member ports up.
If the peer link and keepalive link fails, there could be a chance that both vPC switches are healthy and the failure occurs because of a connectivity issue between the switches. In this situation, both vPC switches claim the primary switch role and keep the vPC member ports up. This situation is known as a split-brain scenario. Because the peer link is no longer available, the two vPC switches cannot synchronize the unicast MAC address and the IGMP group and therefore they cannot maintain the complete unicast and multicast forwarding table. This situation is rare.
We recommend that you have a well-planned network design that includes spreading peer links and keepalive links to multiple ASICs or multiple modules and different cabling routes for keepalive and peer links to avoid a double failure.
So take a look at these options. They show how to: (1) deploy vPC peer-links incorrectly with only one module, (2) deploy vPC peer-links correctly with two separate modules, and (3) deploy vPC peer-links correctly with only one module.
The below diagram is shown with only a single layer-3 interface as the vPC peer link – PLEASE DON’T DO THIS!
Switch # 1
NexusSwitch1(config)# interface e2/1 NexusSwitch1(config-if)# desc Single L3 vPC link NexusSwitch1(config-if)# vrf forwarding VPC NexusSwitch1(config-if)# ip address 192.168.1.1/30 NexusSwitch1(config-if)# vpc peer-link NexusSwitch1(config-if)# exit NexusSwitch1(config)# vpc domain 100 NexusSwitch1(config-vpc-domain)# peer-keepalive destination 192.168.1.2 source 192.168.1.1 vrf VPC
Switch # 2
NexusSwitch2(config)# interface e2/1 NexusSwitch2(config-if)# desc Single L3 vPC link NexusSwitch2(config-if)# vrf forwarding VPC NexusSwitch2(config-if)# ip address 192.168.1.1/30 NexusSwitch2(config-if)# vpc peer-link NexusSwitch2(config-if)# exit NexusSwitch2(config)# vpc domain 100 NexusSwitch2(config-vpc-domain)# peer-keepalive destination 192.168.1.1 source 192.168.1.1 vrf VPC
The below diagram is shown with multiple layer-3 interfaces on a single module as the vPC peer link with a layer-3 port-channel configured- LESS BAD 🙂
Switch # 1
NexusSwitch1(config)# interface e2/1-2 NexusSwitch1(config-if-range)# desc Single L3 vPC link NexusSwitch1(config-if-range)# no switchport NexusSwitch1(config-if-range)# vrf forwarding VPC NexusSwitch1(config-if-range)# channel-group 100 mode none NexusSwitch1(config-if)# exit NexusSwitch1(config)# int Po100 NexusSwitch1(config-if)# desc VPC peer link NexusSwitch1(config-if)# no switchport NexusSwitch1(config-if)# vrf forwarding VPC NexusSwitch1(config-if)# ip address 192.168.1.1/30 NexusSwitch1(config-if)# vpc peer-link NexusSwitch1(config-if)# exit NexusSwitch1(config)# vpc domain 100 NexusSwitch1(config-vpc-domain)# peer-keepalive destination 192.168.1.2 source 192.168.1.1 vrf VPC
Switch # 2
NexusSwitch2(config)# interface e2/1-2 NexusSwitch2(config-if-range)# desc Single L3 vPC link NexusSwitch2(config-if-range)# no switchport NexusSwitch2(config-if-range)# vrf forwarding VPC NexusSwitch2(config-if-range)# channel-group 100 mode none NexusSwitch2(config-if)# exit NexusSwitch2(config)# int Po100 NexusSwitch2(config-if)# desc VPC peer link NexusSwitch2(config-if)# no switchport NexusSwitch2(config-if)# vrf forwarding VPC NexusSwitch2(config-if)# ip address 192.168.1.2/30 NexusSwitch2(config-if)# vpc peer-link NexusSwitch2(config-if)# exit NexusSwitch2(config)# vpc domain 100 NexusSwitch2(config-vpc-domain)# peer-keepalive destination 192.168.1.1 source 192.168.1.2 vrf VPC
The below diagram is shown with multiple layer-3 interfaces on multiple modules as the vPC peer link with a layer-3 port-channel configured- BEST OPTION!
Switch # 1
NexusSwitch1(config)# interface e2/1 NexusSwitch1(config-if)# desc L3 vPC link NexusSwitch1(config-if)# no switchport NexusSwitch1(config-if)# vrf forwarding VPC NexusSwitch1(config-if)# channel-group 100 mode none NexusSwitch1(config-if)# exit NexusSwitch1(config)# interface e3/1 NexusSwitch1(config-if)# desc L3 vPC link NexusSwitch1(config-if)# no switchport NexusSwitch1(config-if)# vrf forwarding VPC NexusSwitch1(config-if)# channel-group 100 mode none NexusSwitch1(config-if)# exit NexusSwitch1(config)# int Po100 NexusSwitch1(config-if)# desc VPC peer link NexusSwitch1(config-if)# no switchport NexusSwitch1(config-if)# vrf forwarding VPC NexusSwitch1(config-if)# ip address 192.168.1.1/30 NexusSwitch1(config-if)# vpc peer-link NexusSwitch1(config-if)# exit NexusSwitch1(config)# vpc domain 100 NexusSwitch1(config-vpc-domain)# peer-keepalive destination 192.168.1.2 source 192.168.1.1 vrf VPC
Switch # 2
NexusSwitch2(config)# interface e2/1 NexusSwitch2(config-if)# desc L3 vPC link NexusSwitch2(config-if)# no switchport NexusSwitch2(config-if)# vrf forwarding VPC NexusSwitch2(config-if)# channel-group 100 mode none NexusSwitch2(config-if)# exit NexusSwitch2(config)# interface e3/1 NexusSwitch2(config-if)# desc L3 vPC link NexusSwitch2(config-if)# no switchport NexusSwitch2(config-if)# vrf forwarding VPC NexusSwitch2(config-if)# channel-group 100 mode none NexusSwitch2(config-if)# exit NexusSwitch2(config)# int Po100 NexusSwitch2(config-if)# desc VPC peer link NexusSwitch2(config-if)# no switchport NexusSwitch2(config-if)# vrf forwarding VPC NexusSwitch2(config-if)# ip address 192.168.1.2/30 NexusSwitch2(config-if)# vpc peer-link NexusSwitch2(config-if)# exit NexusSwitch2(config)# vpc domain 100 NexusSwitch2(config-vpc-domain)# peer-keepalive destination 192.168.1.1 source 192.168.1.2 vrf VPC
