NVIDIA Mellanox MCX5 网卡 DPDK入门 (V20.11)
2023-07-28 08:38:42
昌辉
2598
基于Redhat 8.2，DPDK版本 V20.11 ，MCX512网卡，安装网卡驱动，DPDK编译和安装。赛诺实验室。
1、系统环境检查
[root@node3 ~]# cat /etc/redhat-release Red Hat Enterprise Linux release 8.2 (Ootpa) [root@node3 ~]# uname -r 4.18.0-193.el8.x86_64 [root@node3 ~]# ldd --version ldd (GNU libc) 2.28 [root@node3 ~]# lspci |grep -i Mellanox 03:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5] 03:00.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
2、安装网卡驱动
[root@node3 ~]# ls MLNX_OFED_LINUX-5.4-1.0.3.0-rhel8.2-x86_64.iso MLNX_OFED_LINUX-5.4-1.0.3.0-rhel8.2-x86_64.iso [root@node3 ~]# mount MLNX_OFED_LINUX-5.4-1.0.3.0-rhel8.2-x86_64.iso /mnt mount: /mnt: WARNING: device write-protected, mounted read-only. [root@node3 ~]# cd /mnt/ [root@node3 mnt]# ./mlnxofedinstall --upstream-libs --dpdk Logs dir: /tmp/MLNX_OFED_LINUX.4120.logs General log file: /tmp/MLNX_OFED_LINUX.4120.logs/general.log Verifying KMP rpms compatibility with target kernel... This program will install the MLNX_OFED_LINUX package on your machine. Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed. Those packages are removed due to conflicts with MLNX_OFED_LINUX, do not reinstall them. Do you want to continue?[y/N]:y ...... To load the new driver, run: /etc/init.d/openibd restart [root@node3 mnt]# /etc/init.d/openibd restart Unloading HCA driver: [ OK ] Loading HCA driver and Access Layer: [ OK ]
如果出现提示需要安装相关依赖，直接安装即可，如下：
[root@node3 mnt]# ./mlnxofedinstall --upstream-libs --dpdk Logs dir: /tmp/MLNX_OFED_LINUX.9408.logs General log file: /tmp/MLNX_OFED_LINUX.9408.logs/general.log Verifying KMP rpms compatibility with target kernel... Error: One or more required packages for installing MLNX_OFED_LINUX are missing. Please install the missing packages using your Linux distribution Package Management tool. Run: yum install make tcl tk [root@node3 mnt]# yum install make tcl tk -y
3、安装meson，ninja
[root@node3 ~]# pip3 install meson ninja
4、安装numa库、gcc工具套件
[root@node3 ~]# dnf groupinstall "Development Tools" [root@node3 ~]# yum install -y numactl numactl-devel [root@node3 ~]# dnf install -y gcc-toolset-9 [root@node3 ~]# pip3 install pyelftools
5、下载 编译安装DPDK 20.11
[root@node3 ~]# wget http://fast.dpdk.org/rel/dpdk-20.11.2.tar.xz [root@node3 ~]# tar xf dpdk-20.11.2.tar.xz [root@node3 ~]# cd dpdk-stable-20.11.2/ [root@node3 dpdk-stable-20.11.2]# ls ABI_VERSION buildtools devtools drivers kernel license Makefile meson_options.txt usertools app config doc examples lib MAINTAINERS meson.build README VERSION [root@node3 dpdk-stable-20.11.2]# meson -Dexamples=all build [root@node3 dpdk-stable-20.11.2]# cd build [root@node3 build]# ninja -j8 -v [root@node3 build]# ninja install [root@node3 build]# mkdir -p /mnt/huge [root@node3 build]# mount -t hugetlbfs nodev /mnt/huge [root@node3 build]#
写入fstab
[root@node3 build]# vim /etc/fstab ...... nodev /mnt/huge hugetlbfs defaults 0 0 [root@node3 build]# mount -a
6、安装配置hugepages
[root@node3 build]# yum -y install libhugetlbfs vim /etc/default/grub 将下面行 GRUB_CMDLINE_LINUX="crashkernel=auto resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet" 改为： GRUB_CMDLINE_LINUX="crashkernel=auto resume=/dev/mapper/cl-swap rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet transparent_hugepage=never default_hugepagesz=2M hugepagesz=2M hugepages=1024" 新的内容: transparent_hugepage=never default_hugepagesz=2M hugepagesz=2M hugepages=1024 大页内存每页大小为2M,一共设置1024页面,即2GB,根据自己主机内存情况进行设置,如果内存多的,CPU又支持1GB的大页内存,可以每页设置1GB的大页内存. // 重新生成 grub.cfg [root@node3 build]# grub2-mkconfig -o /boot/grub2/grub.cfg Generating grub configuration file ... done [root@node3 build]# reboot [root@node3 ~]# cat /proc/cmdline |grep hugepages BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-193.el8.x86_64 root=/dev/mapper/rhel-root ro crashkernel=auto resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet transparent_hugepage=never default_hugepagesz=2M hugepagesz=2M hugepages=1024 [root@node3 ~]# cat /proc/meminfo |grep -i HugePages AnonHugePages: 0 kB ShmemHugePages: 0 kB HugePages_Total: 1024 HugePages_Free: 1024 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB [root@node3 ~]# numactl --hardware available: 2 nodes (0-1) node 0 cpus: 0 1 2 3 4 5 6 7 8 9 20 21 22 23 24 25 26 27 28 29 node 0 size: 31790 MB node 0 free: 29135 MB node 1 cpus: 10 11 12 13 14 15 16 17 18 19 30 31 32 33 34 35 36 37 38 39 node 1 size: 32251 MB node 1 free: 30813 MB node distances: node 0 1 0: 10 21 1: 21 10
7、运行 testpmd，使用一根25GbE线缆将网卡2个端口直连。
[root@node3 ~]# lspci |grep -i mellanox 03:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5] 03:00.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
[root@node3 ~]# cd dpdk-stable-20.11.2/build/app/ [root@node3 app]# ./dpdk-testpmd -l 0-4 -n 2 -a 03:00.0 -a 03:00.1 -- -i --rxq=2 --txq=2 --rxd=1024 --txd=1024 --mbcache=512 --burst=64 EAL: Detected 40 lcore(s) EAL: Detected 2 NUMA nodes EAL: Detected static linkage of DPDK EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'PA' EAL: No available hugepages reported in hugepages-1048576kB EAL: Probing VFIO support... EAL: Probe PCI driver: mlx5_pci (15b3:1017) device: 0000:03:00.0 (socket 0) EAL: Probe PCI driver: mlx5_pci (15b3:1017) device: 0000:03:00.1 (socket 0) EAL: No legacy callbacks, legacy socket not created Interactive-mode selected testpmd: create a new mbuf pool : n=212992, size=2176, socket=0 testpmd: preferred mempool ops selected: ring_mp_mc Configuring Port 0 (socket 0) Port 0: 0C:42:A1:7F:3D:C0 Configuring Port 1 (socket 0) Port 1: 0C:42:A1:7F:3D:C1 Checking link statuses... Done testpmd>
