mlx5_common: Failed to initialize global MR share cache

[krish2080]

Hi we are using Mellanox nic 100G interfaces to simulate L2/L3 traffic with our catalyst switches. We have done bare metal installation of TRex v3.04 over centos7.09. We have mapped 100G ports 6 and 8 in trex and trying to start the TRex server and its failing. Can anyone share your inputs/solution for resolving the issue please.

+----+------+---------+-------------------+-------------------------------------------+-----------+------------+----------+
| ID | NUMA | PCI | MAC | Name | Driver | Linux IF | Active |
+====+======+=========+===================+=========================================
| 0 | 0 | 01:00.0 | 48:91:d5:1a:57:22 | Ethernet Controller 10G X550T | ixgbe | eno1 | Active |
+----+------+---------+-------------------+-------------------------------------------+-----------+------------+----------+
| 6 | 0 | 5e:00.0 | b8:3f:d2:4f:56:16 | MT28800 Family [ConnectX-5 Ex] | mlx5_core | enp94s0f0 | |
+----+------+---------+-------------------+-------------------------------------------+-----------+------------+----------+
| 7 | 0 | 5e:00.1 | b8:3f:d2:4f:56:17 | MT28800 Family [ConnectX-5 Ex] | mlx5_core | enp94s0f1 | |
+----+------+---------+-------------------+-------------------------------------------+-----------+------------+----------+
| 8 | 1 | 86:00.0 | b8:3f:d2:e0:94:84 | MT28800 Family [ConnectX-5 Ex] | mlx5_core | enp134s0f0 | |
+----+------+---------+-------------------+-------------------------------------------+-----------+------------+----------+
| 9 | 1 | 86:00.1 | b8:3f:d2:e0:94:85 | MT28800 Family [ConnectX-5 Ex] | mlx5_core | enp134s0f1 | |
+----+------+---------+-------------------+-------------------------------------------+-----------+------------+----------+

./dpdk_nic_bind.py --status

Network devices using DPDK-compatible driver

0000:5e:00.0 'MT28800 Family [ConnectX-5 Ex]' drv=mlx5_core unused=igb_uio,vfio-pci,uio_pci_generic
0000:5e:00.1 'MT28800 Family [ConnectX-5 Ex]' drv=mlx5_core unused=igb_uio,vfio-pci,uio_pci_generic
0000:86:00.0 'MT28800 Family [ConnectX-5 Ex]' drv=mlx5_core unused=igb_uio,vfio-pci,uio_pci_generic
0000:86:00.1 'MT28800 Family [ConnectX-5 Ex]' drv=mlx5_core unused=igb_uio,vfio-pci,uio_pci_generic

Network devices using kernel driver

0000:01:00.0 'Ethernet Controller 10G X550T' if=eno1 drv=ixgbe unused=igb_uio,vfio-pci,uio_pci_generic Active
0000:01:00.1 'Ethernet Controller 10G X550T' if=eno2 drv=ixgbe unused=igb_uio,vfio-pci,uio_pci_generic
0000:19:00.0 'I350 Gigabit Network Connection' if=enp25s0f0 drv=igb unused=igb_uio,vfio-pci,uio_pci_generic
0000:19:00.1 'I350 Gigabit Network Connection' if=enp25s0f1 drv=igb unused=igb_uio,vfio-pci,uio_pci_generic
0000:19:00.2 'I350 Gigabit Network Connection' if=enp25s0f2 drv=igb unused=igb_uio,vfio-pci,uio_pci_generic
0000:19:00.3 'I350 Gigabit Network Connection' if=enp25s0f3 drv=igb unused=igb_uio,vfio-pci,uio_pci_generic
0000:5e:00.0 'MT28800 Family [ConnectX-5 Ex]' if=enp94s0f0 drv=mlx5_core unused=igb_uio,vfio-pci,uio_pci_generic
0000:5e:00.1 'MT28800 Family [ConnectX-5 Ex]' if=enp94s0f1 drv=mlx5_core unused=igb_uio,vfio-pci,uio_pci_generic
0000:86:00.0 'MT28800 Family [ConnectX-5 Ex]' if=enp134s0f0 drv=mlx5_core unused=igb_uio,vfio-pci,uio_pci_generic
0000:86:00.1 'MT28800 Family [ConnectX-5 Ex]' if=enp134s0f1 drv=mlx5_core unused=igb_uio,vfio-pci,uio_pci_generic
0000:af:00.0 'Ethernet Controller XL710 for 40GbE QSFP+' if=enp175s0f0 drv=i40e unused=igb_uio,vfio-pci,uio_pci_generic
0000:af:00.1 'Ethernet Controller XL710 for 40GbE QSFP+' if=enp175s0f1 drv=i40e unused=igb_uio,vfio-pci,uio_pci_generic
0000:d8:00.0 'Ethernet Controller X710 for 10GbE SFP+' if=enp216s0f0 drv=i40e unused=igb_uio,vfio-pci,uio_pci_generic
0000:d8:00.1 'Ethernet Controller X710 for 10GbE SFP+' if=enp216s0f1 drv=i40e unused=igb_uio,vfio-pci,uio_pci_generic
0000:d9:00.0 'Ethernet Controller XL710 for 40GbE QSFP+' if=enp217s0f0 drv=i40e unused=igb_uio,vfio-pci,uio_pci_generic
0000:d9:00.1 'Ethernet Controller XL710 for 40GbE QSFP+' if=enp217s0f1 drv=i40e unused=igb_uio,vfio-pci,uio_pci_generic

Other network devices

0000:d8:00.2 'Ethernet Controller X710 for 10GbE SFP+' unused=i40e,igb_uio,vfio-pci,uio_pci_generic
0000:d8:00.3 'Ethernet Controller X710 for 10GbE SFP+' unused=i40e,igb_uio,vfio-pci,uio_pci_generic

[root@localhost v3.04]# cat /etc/100_cfg1.yaml

Config file generated by dpdk_setup_ports.py

• version: 2
  interfaces: ['5e:00.0', '86:00.0']
  prefix: 100GPORTS
  zmq_pub_port: 4500
  zmq_rpc_port: 4501
  c: 8
  port_info:
  - dest_mac: b8:3f:d2:e0:94:84 # MAC OF LOOPBACK TO IT'S DUAL INTERFACE
  src_mac: b8:3f:d2:4f:56:16
  - dest_mac: b8:3f:d2:4f:56:16 # MAC OF LOOPBACK TO IT'S DUAL INTERFACE
  src_mac: b8:3f:d2:e0:94:84

  platform:
  master_thread_id: 0
  latency_thread_id: 24
  dual_if:
  - socket: 0
  threads: [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20]

    - socket: 1
      threads: [21,22,23,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63]
  

=====================================================================
[root@localhost v3.04]# ./t-rex-64 --cfg /etc/100_cfg1.yaml -i --no-key --no-termio
Starting Scapy server...... Scapy server is started
The ports are bound/configured.
Starting TRex v3.04 please wait ...
mlx5_common: Failed to initialize global MR share cache.
EAL: Requested device 0000:86:00.0 cannot be used
ERROR in DPDK map
Could not find requested interface 86:00.0
Killing Scapy server... Scapy server is killed