NVIDIA Skyway™ GA100 开局配置参考案例
简介
NVIDIA Skyway™ GA100 网关设备主要用于解决Infiniband网络和Ethernet网络之间数据通讯。设备上有8个IB端口和8个以太网口,且端口的网络模式已经被设置,不可变更。
1、开机BIOS设置
Skyway GA100 首次开机需要外接显示器和键盘,设备加电后,键盘按 “Del” 键进入BIOS。选择 “Server Mgmt“,“BMC network configuration” 进入网口IP设置,
给“Lan channel 2” 设置一个静态IP地址,此IP地址是IPMI的接口地址,之后使用网页访问这个IP,用户名和密码默认都是:admin
2、Skyway IPMI管理口连线
上图为Skyway 后面板上的网络接口,红色标记的网口对应BIOS设置的 “Lan channel 2”,此处很容易混淆。绿色标记的网口对应 BIOS 设置的 “Lan channel 1“ ,这个端口也是skyway 操作系统在命令行对应的Mgmt0,也就是我们很熟悉的交换机的带外管理网口。
3、Web登录
要登录到Skyway GA100的带外管理界面(此界面不是交换机Mgmt0的web登录界面),在浏览器输入IPMI的IP地址(192.168.1.101),输入用户名:admin和密码:admin登录。
成功登录后,我们可以启用SOL来操作,进行相关配置命令。
点击 “调用SOL” 按钮后会弹出下载对话框,保存launch.jnlp 这个文件,要打开此文件,本地电脑必须安装Java才能正常打开和运行。双击 launch.jnlp或者右键点击使用Java打开:
由于安全原因,第一次打开下载的文件可能会被java阻止运行,需要把指定的ip访问添加到可信任的“列外站点”列表。
勾选“接受风险”,然后运行
输入密码:admin 点击“确定“
按下“Enter”,弹出像登录交换机一样的命令行界面:
输入用户名:admin 密码:admin
登录交换机:
5、网络规划参考案例
此案例作为基础配置参考,Skyway作为网关设备,IB端口必须接到IB交换机,以太网端口接到以太网交换机,以下图为规划:
6、配置IB交换机(SB7800)
IB交换机在此规划设计中的主要作用是启用 IB子网管理器虚拟化支持,对于本案例只需配置以下命令:
IB-switch-7800 [standalone: master] > enable IB-switch-7800 [standalone: master] # configure terminal IB-switch-7800 [standalone: master] (config) # ib sm IB-switch-7800 [standalone: master] (config) # ib sm virt enable IB-switch-7800 [standalone: master] (config) # ib sm virt-max-ports-in-process 0 IB-switch-7800 [standalone: master] (config) # write memory
7、配置Skyway GA100
gateway-815056 > enable gateway-815056 # configure terminal gateway-815056 (config) # interface ib port-channel 1 ip address 1.1.1.1 /24 gateway-815056 (config) # interface ib port-channel 1 virtual ip address 1.1.1.3 /24 gateway-815056 (config) # interface ethernet port-channel 1 ip address 2.2.2.2 /24 gateway-815056 (config) # ip route 0.0.0.0 0.0.0.0 2.2.2.1 gateway-815056 (config) # write memory
8、配置以太网交换机(SN2010)
onyx1 [standalone: master] > enable onyx1 [standalone: master] # configure terminal onyx1 [standalone: master] (config) # vlan 2 onyx1 [standalone: master] (config vlan 2) # exit onyx1 [standalone: master] (config) # interface vlan 2 onyx1 [standalone: master] (config interface vlan 2) # ip address 2.2.2.1 /24 onyx1 [standalone: master] (config interface vlan 2) # no shutdown onyx1 [standalone: master] (config interface vlan 2) # exit onyx1 [standalone: master] (config) # ip route vrf default 1.1.1.0 /24 2.2.2.2 onyx1 [standalone: master] (config) # interface ethernet 1/22 switchport access vlan 2 onyx1 [standalone: master] (config) # interface port-channel 1 onyx1 [standalone: master] (config interface port-channel 1) # exit onyx1 [standalone: master] (config) # interface ethernet 1/21 channel-group 1 mode active onyx1 [standalone: master] (config) # write memory
9、服务器网卡相关配置
ib网卡:
[root@node4 ~]# cat /etc/sysconfig/network-scripts/ifcfg-ib0 TYPE=InfiniBand BOOTPROTO=none NAME=ib0 DEVICE=ib0 ONBOOT=yes IPADDR=1.1.1.2 NETMASK=255.255.255.0 GATEWAY=1.1.1.1
以太网卡:
[root@node2 ~]# cat /etc/sysconfig/network-scripts/ifcfg-ens1 TYPE=Ethernet BOOTPROTO=none NAME=ens1 DEVICE=ens1 ONBOOT=yes IPADDR=2.2.2.3 NETMASK=255.255.255.0 GATEWAY=2.2.2.1
10、验证
首先检测相关状态是否都正常,再进行连通性测试,带宽测试等操作。
10.1、子网管理器状态是否正常,以及IB子网络是否正常
IB-switch-7800 [standalone: master] # show ib sm enable IB-switch-7800 [standalone: master] # show ib sm virt-max-ports-in-process 0
10.2、Skyway port-channel状态是否正常
gateway-815056 (config) # show interfaces ib port-channel 1 Po1: Admin state : Enabled Operational state : Up Description : GUID : 0014:0500:0a56:5000 MTU : 1500 Active links : 1 X 100G IPv4 address: 1.1.1.1/24 Virtual IPv4 address: 1.1.1.3/24 Broadcast address: 1.1.1.255 Rx: packets : 719970 bytes : 63861760 error packets: 0 Tx: packets : 32262598 bytes : 49548128756 error packets: 0 gateway-815056 # show interfaces ethernet port-channel 1 Po1: Admin state : Enabled Operational state: Up Description : Mac address : B8:CE:F6:68:7A:F3 MTU : 1500 Actual speed : 1 X 100G IPv4 address: 2.2.2.2/24 Broadcast address: 2.2.2.255 Rx: packets : 9724 bytes : 955430 error packets: 0 Tx: packets : 4786 bytes : 505110 error packets: 0
10.3、以太网交换机的port-channel是否正常,相关的ip配置和路由
onyx1 [standalone: master] (config) # show interfaces port-channel 1 Po1: Admin state : Enabled Operational state : Up Description : N/A Mac address : N/A MTU : 1500 bytes (Maximum packet size 1522 bytes) lacp-individual mode: Disabled Flow-control : receive off send off Actual speed : 1 X 100G Width reduction mode: Not supported Switchport mode : access MAC learning mode : Enabled Forwarding mode : inherited cut-through
10.4、网卡状态检查
IB网卡:
[root@node4 ~]# ibdev2netdev mlx5_0 port 1 ==> ib0 (Up) [root@node4 ~]# ifconfig ib0 ib0: flags=4163 mtu 2044 inet 1.1.1.2 netmask 255.255.255.0 broadcast 1.1.1.255 inet6 fe80::e42:a103:b4:d89c prefixlen 64 scopeid 0x20 RX packets 32256173 bytes 49803364919 (46.3 GiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 717535 bytes 43081151 (41.0 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
以太网网卡(Mellanox):
[root@node2 ~]# ibdev2netdev mlx5_0 port 1 ==> ens1 (Up) [root@node2 ~]# ifconfig ens1 ens1: flags=4163 mtu 1500 inet 2.2.2.3 netmask 255.255.255.0 broadcast 2.2.2.255 inet6 fe80::e42:a1ff:feb6:43c6 prefixlen 64 scopeid 0x20 ether 0c:42:a1:b6:43:c6 txqueuelen 1000 (Ethernet) RX packets 717716 bytes 47404900 (45.2 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 32260418 bytes 48838322664 (45.4 GiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
10.5、从ib网卡向以太网卡ping测试
[root@node4 ~]# ping 1.1.1.1 PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data. 64 bytes from 1.1.1.1: icmp_seq=1 ttl=64 time=0.133 ms 64 bytes from 1.1.1.1: icmp_seq=2 ttl=64 time=0.068 ms 64 bytes from 1.1.1.1: icmp_seq=3 ttl=64 time=0.063 ms 64 bytes from 1.1.1.1: icmp_seq=4 ttl=64 time=0.052 ms ^C --- 1.1.1.1 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 68ms rtt min/avg/max/mdev = 0.052/0.079/0.133/0.031 ms [root@node4 ~]# ping 2.2.2.1 PING 2.2.2.1 (2.2.2.1) 56(84) bytes of data. 64 bytes from 2.2.2.1: icmp_seq=1 ttl=64 time=0.144 ms 64 bytes from 2.2.2.1: icmp_seq=2 ttl=64 time=0.178 ms 64 bytes from 2.2.2.1: icmp_seq=3 ttl=64 time=0.170 ms ^C --- 2.2.2.1 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 89ms rtt min/avg/max/mdev = 0.144/0.164/0.178/0.014 ms [root@node4 ~]# ping 2.2.2.2 PING 2.2.2.2 (2.2.2.2) 56(84) bytes of data. 64 bytes from 2.2.2.2: icmp_seq=1 ttl=64 time=0.081 ms 64 bytes from 2.2.2.2: icmp_seq=2 ttl=64 time=0.063 ms 64 bytes from 2.2.2.2: icmp_seq=3 ttl=64 time=0.062 ms ^C --- 2.2.2.2 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 53ms rtt min/avg/max/mdev = 0.062/0.068/0.081/0.012 ms [root@node4 ~]# ping 2.2.2.3 PING 2.2.2.3 (2.2.2.3) 56(84) bytes of data. 64 bytes from 2.2.2.3: icmp_seq=1 ttl=63 time=0.108 ms 64 bytes from 2.2.2.3: icmp_seq=2 ttl=63 time=0.071 ms 64 bytes from 2.2.2.3: icmp_seq=3 ttl=63 time=0.055 ms 64 bytes from 2.2.2.3: icmp_seq=4 ttl=63 time=0.069 ms ^C --- 2.2.2.3 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 76ms rtt min/avg/max/mdev = 0.055/0.075/0.108/0.022 ms
10.6、从以太网卡向ib网卡ping测试
[root@node2 ~]# ping 2.2.2.1 PING 2.2.2.1 (2.2.2.1) 56(84) bytes of data. 64 bytes from 2.2.2.1: icmp_seq=1 ttl=64 time=0.141 ms 64 bytes from 2.2.2.1: icmp_seq=2 ttl=64 time=0.136 ms ^C --- 2.2.2.1 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 999ms rtt min/avg/max/mdev = 0.136/0.138/0.141/0.012 ms [root@node2 ~]# ping 2.2.2.2 PING 2.2.2.2 (2.2.2.2) 56(84) bytes of data. 64 bytes from 2.2.2.2: icmp_seq=1 ttl=63 time=0.068 ms 64 bytes from 2.2.2.2: icmp_seq=2 ttl=63 time=0.056 ms ^C --- 2.2.2.2 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 999ms rtt min/avg/max/mdev = 0.056/0.062/0.068/0.006 ms [root@node2 ~]# ping 1.1.1.1 PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data. 64 bytes from 1.1.1.1: icmp_seq=1 ttl=63 time=0.073 ms From 2.2.2.1 icmp_seq=1 Redirect Host(New nexthop: 2.2.2.2) From 2.2.2.1: icmp_seq=1 Redirect Host(New nexthop: 2.2.2.2) 64 bytes from 1.1.1.1: icmp_seq=2 ttl=63 time=0.052 ms From 2.2.2.1 icmp_seq=2 Redirect Host(New nexthop: 2.2.2.2) From 2.2.2.1: icmp_seq=2 Redirect Host(New nexthop: 2.2.2.2) 64 bytes from 1.1.1.1: icmp_seq=3 ttl=63 time=0.050 ms From 2.2.2.1 icmp_seq=3 Redirect Host(New nexthop: 2.2.2.2) From 2.2.2.1: icmp_seq=3 Redirect Host(New nexthop: 2.2.2.2) ^C --- 1.1.1.1 ping statistics --- 3 packets transmitted, 3 received, +3 errors, 0% packet loss, time 2000ms rtt min/avg/max/mdev = 0.050/0.058/0.073/0.012 ms [root@node2 ~]# ping 1.1.1.2 PING 1.1.1.2 (1.1.1.2) 56(84) bytes of data. 64 bytes from 1.1.1.2: icmp_seq=1 ttl=63 time=0.116 ms From 2.2.2.1 icmp_seq=1 Redirect Host(New nexthop: 2.2.2.2) From 2.2.2.1: icmp_seq=1 Redirect Host(New nexthop: 2.2.2.2) 64 bytes from 1.1.1.2: icmp_seq=2 ttl=63 time=0.085 ms From 2.2.2.1 icmp_seq=2 Redirect Host(New nexthop: 2.2.2.2) From 2.2.2.1: icmp_seq=2 Redirect Host(New nexthop: 2.2.2.2) 64 bytes from 1.1.1.2: icmp_seq=3 ttl=63 time=0.089 ms From 2.2.2.1 icmp_seq=3 Redirect Host(New nexthop: 2.2.2.2) From 2.2.2.1: icmp_seq=3 Redirect Host(New nexthop: 2.2.2.2) ^C --- 1.1.1.2 ping statistics --- 3 packets transmitted, 3 received, +3 errors, 0% packet loss, time 1999ms rtt min/avg/max/mdev = 0.085/0.096/0.116/0.017 ms
与 NVIDIA产品相关的图片或视频(完整或部分)的版权均归 NVIDIA Corporation 所有。