cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
Soap1
Punch Card
Posts: 32
Registered: ‎03-29-2017
Location: US
Views: 299
Message 11 of 20

Re: EN4093R - Configure similar to a Cisco FEX Module?

Is this the expected STP setting with this configuration? My upstream devices are Nexus 5K and my blades are a mixture of vSphere leveraging vDS and/or Windows/Linux hosts that will tag network traffic.

 

!
interface port EXT1
	switchport access vlan 4090
	tagpvid-ingress
	exit
!
interface port EXT2
	switchport access vlan 4090
	tagpvid-ingress
	exit
!
vlan 4090
	name "Tunnel"
!
!
spanning-tree stp 26 vlan 4090
!
!
interface port EXT1
	lacp mode active
	lacp key 1000
!
interface port EXT2
	lacp mode active
	lacp key 1000
!
failover enable
failover trigger 1 amon admin-key 1000
failover trigger 1 enable
!

 

I am also curious on the recommended physical connectivity between the EN4093 and the Cisco 5K. (static port channel vs lacp) We have two uplinks (EXT1/2) per EN4093 going back to 5K(a) and 5K(b) switches.

 

Lenovo Employee mslavin
Lenovo Employee
Posts: 224
Registered: ‎03-31-2015
Location: US
Views: 282
Message 12 of 20

Re: EN4093R - Configure similar to a Cisco FEX Module?

Hello,

 

For spanning-tree, what you show would be the default STP config. Since you are building loop free, you could also choose to disable STP outright on the EN4093 (spanning-tree mode disable). If you do disable STP on the Flex, you need to remember to enable edge trunk on the Cisco ports facing the Flex switches (spanning-tree port type edge trunk). And if you leave STP enabled on the Flex, you need to remember to set the spanning-tree port type to normal (spanning-tree port type normal) (to avoid issues with Cisco Bridge Assurance).

 

On a non-STP-related comment, when using this sort of VLAN agnostic design, I recommend to set the native on the upstream Cisco’s facing the Flex to some unused value, or otherwise do not allow or use it. And then use only tagged traffic for all communications between 5K’s and Flex hosts. Note this is not a “must”, just a good idea, as using an untagged VLAN/native could cause issues if one Flex host wants to use a different untagged VLAN than another Flex host. Just tag everything and it will work fine.

 

Physical connectivity in my mind is always getting the most robust, best performing design under whatever constraints exist. For example, I would recommend a full mesh vPC/vLAG design from the Flex to the 5Ks, as being most robust/highest performing. Baring that (if you do not want to use full mesh) I would recommend still running vPC on the Cisco side, one each for each Flex switch. That would be shown on page 5 of the design guide I had previously attached to this thread. And if you want to keep it as simple as possible, I would go with the inverted U as seen on page 6 of that design guide (The inverted U was the most common design for Flex switches before vendors started coming out with cross-switch aggregations, like vPC and vLAG).

 

As to LACP or static, that is almost a religious discussion as to which is better. I have seen cases were one has worked better than the other, but in general, I recommend LACP over static. This gets back to the kinds of issues that can trip up LACP (i.e. bugs in code, heavy pause frame environment stopping LACPDU’s, etc) are much less likely to occur than the issues that trip up static aggregation (someone mis-cabling or mis-configuring ports). Thus my recommendation of LACP.

 

Hope this helps.

 

Thanks, Matt

 

Soap1
Punch Card
Posts: 32
Registered: ‎03-29-2017
Location: US
Views: 268
Message 13 of 20

Re: EN4093R - Configure similar to a Cisco FEX Module?

Thanks Matt, I am trying to get some time today or tomorrow to sit with my network team to bounce the designs off of them. I will report back and mark the relevent post as an accepted solution, just holding off as I may have a few more questions along the way.

Soap1
Punch Card
Posts: 32
Registered: ‎03-29-2017
Location: US
Views: 228
Message 14 of 20

Re: EN4093R - Configure similar to a Cisco FEX Module?

So with the switch configured I see that 'sw1' shows interfaces online, but 'sw2' shows them down. If I reboot 'sw1' the interfaces on 'sw2' come up, only until 'sw1' comes back online which then causes everything to flip back to it. is this the expected behavior? I guess I totally missed that if so and was anticipating both switches to be online active/active.

 

-----------------------------------------------------------------------
Alias   Port   Speed    Duplex     Flow Ctrl      Link     Description 
------- ----   -----   --------  --TX-----RX--   ------   -------------
INTA1    1     1G/10G    full     yes    yes    disabled     INTA1
INTA2    2     1G/10G    full     yes    yes    disabled     INTA2
INTA3    3     1G/10G    full     yes    yes    disabled     INTA3
INTA4    4     1G/10G    full     yes    yes    disabled     INTA4
INTA5    5     1G/10G    full     yes    yes    disabled     INTA5
INTA6    6     1G/10G    full     yes    yes    disabled     INTA6
INTA7    7     1G/10G    full     yes    yes    disabled     INTA7
INTA8    8     1G/10G    full     yes    yes    disabled     INTA8
INTA9    9     1G/10G    full     yes    yes    disabled     INTA9
INTA10   10    1G/10G    full     yes    yes    disabled     INTA10
INTA11   11    1G/10G    full     yes    yes    disabled     INTA11
INTA12   12    1G/10G    full     yes    yes    disabled     INTA12
INTA13   13    1G/10G    full     yes    yes    disabled     INTA13
INTA14   14    1G/10G    full     yes    yes    disabled     INTA14
EXT1     43    10000     full      no     no       up        EXT1
EXT2     44    10000     full      no     no       up        EXT2
EXT3     45    10000     full      no     no      down       EXT3
EXT4     46    10000     full      no     no      down       EXT4
EXT5     47    10000     full      no     no      down       EXT5
EXT6     48    10000     full      no     no      down       EXT6
EXT7     49    10000     full      no     no      down       EXT7
EXT8     50    10000     full      no     no      down       EXT8
EXT9     51    1G/10G    full      no     no      down       EXT9
EXT10    52    1G/10G    full      no     no      down       EXT10
EXTM     65      any     auto     yes    yes      down       EXTM
MGT1     66     1000     full      no     no       up        MGT1
hhlchasa801_sw2#
Aug 13 05:50:16 hhlchasa801_sw2 NOTICE  lacp: LACP is up on port EXT1

Aug 13 05:50:16 hhlchasa801_sw2 NOTICE  failover: Trigger 1 is up, control ports are auto controlled.

Aug 13 05:50:16 hhlchasa801_sw2 NOTICE  lacp: LACP is up on port EXT2

Aug 13 05:50:16 hhlchasa801_sw2 NOTICE  server: link up on port INTA1

Aug 13 05:50:16 hhlchasa801_sw2 NOTICE  server: link up on port INTA2

Aug 13 05:50:16 hhlchasa801_sw2 NOTICE  server: link up on port INTA3

Aug 13 05:50:16 hhlchasa801_sw2 NOTICE  server: link up on port INTA6

Aug 13 05:50:16 hhlchasa801_sw2 NOTICE  server: link up on port INTA4

Aug 13 05:50:16 hhlchasa801_sw2 NOTICE  server: link up on port INTA12

Aug 13 05:50:16 hhlchasa801_sw2 NOTICE  server: link up on port INTA14
show interface link
-----------------------------------------------------------------------
Alias   Port   Speed    Duplex     Flow Ctrl      Link     Description 
------- ----   -----   --------  --TX-----RX--   ------   -------------
INTA1    1     10000     full     yes    yes       up        INTA1
INTA2    2     10000     full     yes    yes       up        INTA2
INTA3    3     10000     full     yes    yes       up        INTA3
INTA4    4     10000     full     yes    yes       up        INTA4
INTA5    5     1G/10G    full     yes    yes      down       INTA5
INTA6    6     10000     full     yes    yes       up        INTA6
INTA7    7     1G/10G    full     yes    yes      down       INTA7
INTA8    8     1G/10G    full     yes    yes      down       INTA8
INTA9    9     1G/10G    full     yes    yes      down       INTA9
INTA10   10    1G/10G    full     yes    yes      down       INTA10
INTA11   11    1G/10G    full     yes    yes      down       INTA11
INTA12   12    10000     full     yes    yes       up        INTA12
INTA13   13    1G/10G    full     yes    yes      down       INTA13
INTA14   14    10000     full     yes    yes       up        INTA14
EXT1     43    10000     full      no     no       up        EXT1
EXT2     44    10000     full      no     no       up        EXT2
EXT3     45    10000     full      no     no      down       EXT3
EXT4     46    10000     full      no     no      down       EXT4
EXT5     47    10000     full      no     no      down       EXT5
EXT6     48    10000     full      no     no      down       EXT6
EXT7     49    10000     full      no     no      down       EXT7
EXT8     50    10000     full      no     no      down       EXT8
EXT9     51    1G/10G    full      no     no      down       EXT9
EXT10    52    1G/10G    full      no     no      down       EXT10
EXTM     65      any     auto     yes    yes      down       EXTM
MGT1     66     1000     full      no     no       up        MGT1
hhlchasa801_sw2#
hhlchasa801_sw2#
hhlchasa801_sw2#
Aug 13 05:52:04 hhlchasa801_sw2 NOTICE  link: link down on port EXT1

Aug 13 05:52:04 hhlchasa801_sw2 NOTICE  lacp: LACP is down on port EXT1

Aug 13 05:52:04 hhlchasa801_sw2 NOTICE  link: link down on port EXT2

Aug 13 05:52:04 hhlchasa801_sw2 NOTICE  lacp: LACP is down on port EXT2

Aug 13 05:52:04 hhlchasa801_sw2 WARNING failover: Trigger 1 is down, control ports are auto disabled.

Aug 13 05:52:04 hhlchasa801_sw2 NOTICE  server: link down on port INTA1

Aug 13 05:52:04 hhlchasa801_sw2 NOTICE  server: link down on port INTA2

Aug 13 05:52:04 hhlchasa801_sw2 NOTICE  server: link down on port INTA3

Aug 13 05:52:04 hhlchasa801_sw2 NOTICE  server: link down on port INTA4

Aug 13 05:52:04 hhlchasa801_sw2 NOTICE  server: link down on port INTA6

Aug 13 05:52:04 hhlchasa801_sw2 NOTICE  server: link down on port INTA12

Aug 13 05:52:04 hhlchasa801_sw2 NOTICE  server: link down on port INTA14

Aug 13 05:54:30 hhlchasa801_sw2 ALERT   system: Protocol control discards: arp-bcast or ipv6-nd  packets are received at rate higher than 200pps,hence are discarded on queue 5!
Lenovo Employee mslavin
Lenovo Employee
Posts: 224
Registered: ‎03-31-2015
Location: US
Views: 222
Message 15 of 20

Re: EN4093R - Configure similar to a Cisco FEX Module?

Hello,

 

When properly configured the uplinks on both switches should be up. Based on what I am seeing in your logs, I am betting the upstream switches are misconfigured and are taking one side of the Flex down when both sides try to come up. For example, if the upstream switch has been configured for a single 4 port aggregation facing the Flex, but the switches are NOT in vLAG and are operating as two independent switches (two x 2 port aggregations on the Flex side). In that case, and depending on the code running on the upstream switch, it may choose to err-disable one side or the other when it sees them coming up as two separate LACP aggregations. The Flex switches are just reacting to what they are seeing based on their configs (they have failover configured, and from your log, the uplinks are being taken down by the OTHER side, and thus failover on the Flex switch is kicking in and shutting down the INT ports to prevent a black hole).

 

Short answer, look at the upstream switches and tell us exactly how they are configured.

 

Also look in the logs of the upstream switches it should tell you WHY it took the ports down (and I'm betting you will see it took them down for reasons as noted above, split aggregation on the Flex side, when the Cisco side is configured for a single 4 port aggregation). And if so, you need to either convert the Flex to vLAG and a single 4 port Agg, or convert the upstream to two independent LACP aggregations, one each for each Flex switch..

 

Please let us know what you see on the upstream switches.

 

Thanks, Matt

 

Soap1
Punch Card
Posts: 32
Registered: ‎03-29-2017
Location: US
Views: 197
Message 16 of 20

Re: EN4093R - Configure similar to a Cisco FEX Module?

My network team wants to know that if they have our upstream 5K switches VPC'd together would that be a problem with this proposed configuration? There is some concern on our side on the possibility of creating a loop I believe.

Highlighted
Soap1
Punch Card
Posts: 32
Registered: ‎03-29-2017
Location: US
Views: 190
Message 17 of 20

Re: EN4093R - Configure similar to a Cisco FEX Module?

Actually, we are going to switch over to design #1 on the design diagram you had provided it looks like now as shown below. Will this still allow me to use the 'tunnel' setup we have been previously discussing and if so can you direct me on what this new config would look like?

 

Flex_Config.jpg

Lenovo Employee mslavin
Lenovo Employee
Posts: 224
Registered: ‎03-31-2015
Location: US
Views: 148
Message 18 of 20

Re: EN4093R - Configure similar to a Cisco FEX Module?

Hello,

 

Either design will not create a loop. But the first design is the best option. I have attached an PDF file that provides an example of the configs to produce design 1 in a VLAN agnostic mode.

 

Let me know if you have any questions.

 

Thanks, Matt

 

Soap1
Punch Card
Posts: 32
Registered: ‎03-29-2017
Location: US
Views: 142
Message 19 of 20

Re: EN4093R - Configure similar to a Cisco FEX Module?

Trying to understand this below piece in relation to the switches. The diagrams don't show this but is this just for the connecting between both EN4093 'MGMT' ports?

 

what is 'int ip 127'

 

! configure EXTM for vLAG healthcheck
int ip 127
 ip add 10.10.10.1 255.255.255.252 enable
Lenovo Employee mslavin
Lenovo Employee
Posts: 224
Registered: ‎03-31-2015
Location: US
Views: 137
Message 20 of 20

Re: EN4093R - Configure similar to a Cisco FEX Module?

Hello,

 

int ip 127 is hard tied to the EXTM RJ45 port on the rear of the switch, that we typically use for the vLAG health check. By assigning an IP to int ip 127 you are assigning it to EXTM. You will also want to run a cross connect between EXTM ports on the switch, as the health check path for vLAG. Note the only purpose of the vLAG health check is to monitor if both of the switches are up if the ISL goes down. If the ISL goes down, and health  check is up, than the "secondary" vLAG switch err-disables any vLAG aggregations ,to prevent black holing from a split brain scenario that would happen without an active ISL link.

 

Thanks, Matt

 

Check out current deals!


Shop current deals

Top Kudoed Authors