NVIDIA CumulusLinux 交换机配置L2 MLAG
2023-07-28 16:26:13
昌辉
1206
NVIDIA® Cumulus® Linux 交换机在生产环境经常会遇到2种常见的使用场景,
2层接入
数据中心接入(基于BGP unnumbered 组网)
对于超融合业务和高可用业务,则会用到MLAG技术(友商称之为“堆叠”)
要求
配置MLAG有以下要求:
必须是2台交换机
2台交换机系统版本要一致
2台交换机之间必须直连至少2根高带宽线缆(比如2根 100GbE)
2层接入,参考案例
2层接入的组网和配置是最简单的,下面是组网和常用的基础数据规划参考。
cumulusA | cumulusB | |
eth0 | 192.168.1.204 | 192.168.1.205 |
peerlink interface | 1/17,1/18 | 1/17,1/18 |
MLAG interface for L3 switch | 1/20 | 1/20 |
MLAG interface for server | 1/1 | 1/1 |
No MLAG interface | 1/2 | 1/2 |
配置步骤
配置mlag
cumulusA
net add clag peer sys-mac 44:38:39:FF:00:AA interface swp17-18 primary backup-ip 192.168.1.205
cumulusB
net add clag peer sys-mac 44:38:39:FF:00:AA interface swp17-18 secondary backup-ip 192.168.1.204
配置bond组
net add bridge bridge vids 1-100 net add bridge bridge pvid 5 net add bridge stp treeprio 4096 net add clag port bond BOND1 interface swp1 clag-id 1 net add clag port bond BOND20 interface swp20 clag-id 20
查看clag状态
cumulus@c2:mgmt:~$ net show clag The peer is alive Our Priority, ID, and Role: 2000 1c:34:da:96:1b:c2 secondary Peer Priority, ID, and Role: 1000 1c:34:da:96:16:c2 primary Peer Interface and IP: peerlink.4094 fe80::1e34:daff:fe96:16c2 (linklocal) Backup IP: 192.168.1.204 (inactive) System MAC: 44:38:39:ff:00:aa CLAG Interfaces Our Interface Peer Interface CLAG Id Conflicts Proto-Down Reason ---------------- ---------------- ------- -------------------- ----------------- BOND1 BOND1 1 - - BOND20 BOND20 20 - -
【可选】配置一个接服务器node4主备模式的端口
net add bridge bridge ports swp2 net add interface swp2 bridge access 5
在对应BOND1端口的服务器node1网卡配置bonding,mode=4,网卡bonding配置参考链接
查看交换机的BOND1状态
cumulusA
cumulus@c1:mgmt:~$ net show interface bonds Name Speed MTU Mode Summary -- -------- ----- ---- ------- ---------------------------------- UP BOND1 25G 9216 802.3ad Bond Members: swp1(UP) UP BOND20 100G 9216 802.3ad Bond Members: swp20(UP) UP peerlink 50G 9216 802.3ad Bond Members: swp17(UP), swp18(UP)
cumulusB
cumulus@c2:mgmt:~$ net show interface bonds Name Speed MTU Mode Summary -- -------- ----- ---- ------- ---------------------------------- UP BOND1 25G 9216 802.3ad Bond Members: swp1(UP) UP BOND20 100G 9216 802.3ad Bond Members: swp20(UP) UP peerlink 50G 9216 802.3ad Bond Members: swp17(UP), swp18(UP)
查看服务器node1的bonding状态
[root@node1 ~]# cat /proc/net/bonding/bond1 Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011) Bonding Mode: IEEE 802.3ad Dynamic link aggregation Transmit Hash Policy: layer2 (0) MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 Peer Notification Delay (ms): 0 802.3ad info LACP rate: slow Min links: 0 Aggregator selection policy (ad_select): stable System priority: 65535 System MAC address: 0c:42:a1:a0:e0:70 Active Aggregator Info: Aggregator ID: 1 Number of ports: 2 Actor Key: 21 Partner Key: 21 Partner Mac Address: 44:38:39:ff:00:aa Slave Interface: ens1f0 MII Status: up Speed: 25000 Mbps Duplex: full Link Failure Count: 4 Permanent HW addr: 0c:42:a1:a0:e0:70 Slave queue ID: 0 Aggregator ID: 1 Actor Churn State: none Partner Churn State: none Actor Churned Count: 2 Partner Churned Count: 2 details actor lacp pdu: system priority: 65535 system mac address: 0c:42:a1:a0:e0:70 port key: 21 port priority: 255 port number: 1 port state: 61 details partner lacp pdu: system priority: 65535 system mac address: 44:38:39:ff:00:aa oper key: 21 port priority: 255 port number: 1 port state: 63 Slave Interface: ens1f1 MII Status: up Speed: 25000 Mbps Duplex: full Link Failure Count: 2 Permanent HW addr: 0c:42:a1:a0:e0:71 Slave queue ID: 0 Aggregator ID: 1 Actor Churn State: none Partner Churn State: none Actor Churned Count: 2 Partner Churned Count: 1 details actor lacp pdu: system priority: 65535 system mac address: 0c:42:a1:a0:e0:70 port key: 21 port priority: 255 port number: 2 port state: 61 details partner lacp pdu: system priority: 65535 system mac address: 44:38:39:ff:00:aa oper key: 21 port priority: 255 port number: 1 port state: 63
在服务器node1使用ethtool查看bond1接口状态
[root@node1 ~]# ethtool bond1 Settings for bond1: Supported ports: [ ] Supported link modes: Not reported Supported pause frame use: No Supports auto-negotiation: No Supported FEC modes: Not reported Advertised link modes: Not reported Advertised pause frame use: No Advertised auto-negotiation: No Advertised FEC modes: Not reported Speed: 50000Mb/s Duplex: Full Port: Other PHYAD: 0 Transceiver: internal Auto-negotiation: off Link detected: yes
【可选】查看服务器node4的bonding状态
[root@node4 ~]# cat /proc/net/bonding/bond1 Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011) Bonding Mode: fault-tolerance (active-backup) Primary Slave: None Currently Active Slave: ens1f1 MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 Peer Notification Delay (ms): 0 Slave Interface: ens1f0 MII Status: up Speed: 25000 Mbps Duplex: full Link Failure Count: 0 Permanent HW addr: 0c:42:a1:7f:3d:c0 Slave queue ID: 0 Slave Interface: ens1f1 MII Status: up Speed: 25000 Mbps Duplex: full Link Failure Count: 0 Permanent HW addr: 0c:42:a1:7f:3d:c1 Slave queue ID: 0
node1到node4 的ping测试
[root@node1 ~]# ping 172.30.1.14 PING 172.30.1.14 (172.30.1.14) 56(84) bytes of data. 64 bytes from 172.30.1.14: icmp_seq=1 ttl=64 time=0.156 ms 64 bytes from 172.30.1.14: icmp_seq=2 ttl=64 time=0.123 ms 64 bytes from 172.30.1.14: icmp_seq=3 ttl=64 time=0.120 ms 64 bytes from 172.30.1.14: icmp_seq=4 ttl=64 time=0.137 ms ^C --- 172.30.1.14 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 69ms rtt min/avg/max/mdev = 0.120/0.134/0.156/0.014 ms
node4到node1的ping测试
[root@node4 ~]# ping 172.30.1.11 PING 172.30.1.11 (172.30.1.11) 56(84) bytes of data. 64 bytes from 172.30.1.11: icmp_seq=1 ttl=64 time=0.295 ms 64 bytes from 172.30.1.11: icmp_seq=2 ttl=64 time=0.151 ms 64 bytes from 172.30.1.11: icmp_seq=3 ttl=64 time=0.143 ms 64 bytes from 172.30.1.11: icmp_seq=4 ttl=64 time=0.151 ms ^C --- 172.30.1.11 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 54ms rtt min/avg/max/mdev = 0.143/0.185/0.295/0.063 ms
查看mac地址表
cumulus@c1:mgmt:~$ net show bridge macs VLAN Master Interface MAC TunnelDest State Flags LastSeen -------- ------ --------- ----------------- ---------- --------- ----- -------- 5 bridge BOND1 0c:42:a1:a0:e0:70 00:01:22 5 bridge peerlink 0c:42:a1:7f:3d:c0 00:00:22 untagged bridge BOND1 1c:34:da:96:16:ec permanent 00:33:53 untagged bridge peerlink 1c:34:da:96:16:c2 permanent 00:40:34 untagged bridge swp2 1c:34:da:96:16:e9 permanent 00:20:59
与 NVIDIA产品相关的图片或视频(完整或部分)的版权均归 NVIDIA Corporation 所有。