我运行了一个小型集群进行mpi计算，最近我们购买了一些EDR Infiniband设备。我正在用两台计算机进行测试，通过一个非托管交换机连接，我能够在两个节点中运行一个有30个进程的测试程序。监控Infiband计数器在
/sys/class/infiniband/mlx5_0/ports/1/counters
现在我想用以太网TCP连接运行同样的测试，以获得一些关于Inifiniband提供的真正改进的数据，但我无法使其工作。我已经尝试了所有类型的mca参数，如。
/usr/lib64/openmpi3/bin/mpirun --hostfile hostfile.txt --mca btl ^openib -np 30 nek5000
/usr/lib64/openmpi3/bin/mpirun --hostfile hostfile.txt --mca btl tcp,self,vader -np 30 nek5000
/usr/lib64/openmpi3/bin/mpirun --hostfile hostfile.txt --mca btl tcp,self,vader --mca btl_openib_if_exclude mlx5_0 -np 30 nek5000
都没有成功 当程序运行时，Infiniband的计数器在增加，运行时间与Infiniband的没有区别。两台电脑都运行Centos 7，这是ompi_info的结果。
Package: Open MPI [email protected]
Distribution
Open MPI: 3.1.3
Open MPI repo revision: v3.1.3
Open MPI release date: Oct 29, 2018
Open RTE: 3.1.3
Open RTE repo revision: v3.1.3
Open RTE release date: Oct 29, 2018
OPAL: 3.1.3
OPAL repo revision: v3.1.3
OPAL release date: Oct 29, 2018
MPI API: 3.1.0
Ident string: 3.1.3
Prefix: /usr/lib64/openmpi3
Configured architecture: x86_64-unknown-linux-gnu
Configure host: x86-02.bsys.centos.org
Configured by: mockbuild
Configured on: Thu Aug 22 16:51:48 UTC 2019
Configure host: x86-02.bsys.centos.org
Configure command line: '--prefix=/usr/lib64/openmpi3'
'--mandir=/usr/share/man/openmpi3-x86_64'
'--includedir=/usr/include/openmpi3-x86_64'
'--sysconfdir=/etc/openmpi3-x86_64'
'--disable-silent-rules' '--enable-builtin-atomics'
'--enable-mpi-cxx' '--with-sge' '--with-valgrind'
'--enable-memchecker' '--with-hwloc=/usr'
'--with-ucx' 'CC=gcc' 'CXX=g++'
'LDFLAGS=-Wl,-z,relro ' 'CFLAGS= -O2 -g -pipe -Wall
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions
-fstack-protector-strong --param=ssp-buffer-size=4
-grecord-gcc-switches -m64 -mtune=generic'
'CXXFLAGS= -O2 -g -pipe -Wall
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions
-fstack-protector-strong --param=ssp-buffer-size=4
-grecord-gcc-switches -m64 -mtune=generic'
'FC=gfortran' 'FCFLAGS= -O2 -g -pipe -Wall
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions
-fstack-protector-strong --param=ssp-buffer-size=4
-grecord-gcc-switches -m64 -mtune=generic'
Built by: mockbuild
Built on: Thu Aug 22 16:55:56 UTC 2019
Built host: x86-02.bsys.centos.org
C bindings: yes
C++ bindings: yes
Fort mpif.h: yes (all)
Fort use mpi: yes (limited: overloading)
Fort use mpi size: deprecated-ompi-info-value
Fort use mpi_f08: no
Fort mpi_f08 compliance: The mpi_f08 module was not built
Fort mpi_f08 subarrays: no
Java bindings: no
Wrapper compiler rpath: runpath
C compiler: gcc
C compiler absolute: /usr/bin/gcc
C compiler family name: GNU
C compiler version: 4.8.5
C++ compiler: g++
C++ compiler absolute: /usr/bin/g++
Fort compiler: gfortran
Fort compiler abs: /usr/bin/gfortran
Fort ignore TKR: no
Fort 08 assumed shape: no
Fort optional args: no
Fort INTERFACE: yes
Fort ISO_FORTRAN_ENV: yes
Fort STORAGE_SIZE: no
Fort BIND(C) (all): no
Fort ISO_C_BINDING: yes
Fort SUBROUTINE BIND(C): no
Fort TYPE,BIND(C): no
Fort T,BIND(C,name="a"): no
Fort PRIVATE: no
Fort PROTECTED: no
Fort ABSTRACT: no
Fort ASYNCHRONOUS: no
Fort PROCEDURE: no
Fort USE...ONLY: no
Fort C_FUNLOC: no
Fort f08 using wrappers: no
Fort MPI_SIZEOF: no
C profiling: yes
C++ profiling: yes
Fort mpif.h profiling: yes
Fort use mpi profiling: yes
Fort use mpi_f08 prof: no
C++ exceptions: no
Thread support: posix (MPI_THREAD_MULTIPLE: yes, OPAL support: yes,
OMPI progress: no, ORTE progress: yes, Event lib:
yes)
Sparse Groups: no
Internal debug support: no
MPI interface warnings: yes
MPI parameter check: runtime
Memory profiling support: no
Memory debugging support: no
dl support: yes
Heterogeneous support: no
mpirun default --prefix: no
MPI_WTIME support: native
Symbol vis. support: yes
Host topology support: yes
MPI extensions: affinity, cuda
FT Checkpoint support: no (checkpoint thread: no)
C/R Enabled Debugging: no
MPI_MAX_PROCESSOR_NAME: 256
MPI_MAX_ERROR_STRING: 256
MPI_MAX_OBJECT_NAME: 64
MPI_MAX_INFO_KEY: 36
MPI_MAX_INFO_VAL: 256
MPI_MAX_PORT_NAME: 1024
MPI_MAX_DATAREP_STRING: 128
MCA allocator: basic (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA allocator: bucket (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA backtrace: execinfo (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA btl: openib (MCA v2.1.0, API v3.0.0, Component v3.1.3)
MCA btl: self (MCA v2.1.0, API v3.0.0, Component v3.1.3)
MCA btl: tcp (MCA v2.1.0, API v3.0.0, Component v3.1.3)
MCA btl: usnic (MCA v2.1.0, API v3.0.0, Component v3.1.3)
MCA btl: vader (MCA v2.1.0, API v3.0.0, Component v3.1.3)
MCA compress: bzip (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA compress: gzip (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA crs: none (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA dl: dlopen (MCA v2.1.0, API v1.0.0, Component v3.1.3)
MCA event: libevent2022 (MCA v2.1.0, API v2.0.0, Component
v3.1.3)
MCA hwloc: external (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA if: linux_ipv6 (MCA v2.1.0, API v2.0.0, Component
v3.1.3)
MCA if: posix_ipv4 (MCA v2.1.0, API v2.0.0, Component
v3.1.3)
MCA installdirs: env (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA installdirs: config (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA memchecker: valgrind (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA memory: patcher (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA mpool: hugepage (MCA v2.1.0, API v3.0.0, Component v3.1.3)
MCA patcher: overwrite (MCA v2.1.0, API v1.0.0, Component
v3.1.3)
MCA pmix: flux (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA pmix: isolated (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA pmix: pmix2x (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA pstat: linux (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA rcache: grdma (MCA v2.1.0, API v3.3.0, Component v3.1.3)
MCA reachable: weighted (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA shmem: mmap (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA shmem: posix (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA shmem: sysv (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA timer: linux (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA dfs: app (MCA v2.1.0, API v1.0.0, Component v3.1.3)
MCA dfs: orted (MCA v2.1.0, API v1.0.0, Component v3.1.3)
MCA dfs: test (MCA v2.1.0, API v1.0.0, Component v3.1.3)
MCA errmgr: default_app (MCA v2.1.0, API v3.0.0, Component
v3.1.3)
MCA errmgr: default_hnp (MCA v2.1.0, API v3.0.0, Component
v3.1.3)
MCA errmgr: default_orted (MCA v2.1.0, API v3.0.0, Component
v3.1.3)
MCA errmgr: default_tool (MCA v2.1.0, API v3.0.0, Component
v3.1.3)
MCA errmgr: dvm (MCA v2.1.0, API v3.0.0, Component v3.1.3)
MCA ess: env (MCA v2.1.0, API v3.0.0, Component v3.1.3)
MCA ess: hnp (MCA v2.1.0, API v3.0.0, Component v3.1.3)
MCA ess: pmi (MCA v2.1.0, API v3.0.0, Component v3.1.3)
MCA ess: singleton (MCA v2.1.0, API v3.0.0, Component
v3.1.3)
MCA ess: slurm (MCA v2.1.0, API v3.0.0, Component v3.1.3)
MCA ess: tool (MCA v2.1.0, API v3.0.0, Component v3.1.3)
MCA filem: raw (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA grpcomm: direct (MCA v2.1.0, API v3.0.0, Component v3.1.3)
MCA iof: hnp (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA iof: orted (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA iof: tool (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA notifier: syslog (MCA v2.1.0, API v1.0.0, Component v3.1.3)
MCA odls: default (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA oob: tcp (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA oob: ud (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA plm: isolated (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA plm: rsh (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA plm: slurm (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA ras: gridengine (MCA v2.1.0, API v2.0.0, Component
v3.1.3)
MCA ras: simulator (MCA v2.1.0, API v2.0.0, Component
v3.1.3)
MCA ras: slurm (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA regx: fwd (MCA v2.1.0, API v1.0.0, Component v3.1.3)
MCA regx: reverse (MCA v2.1.0, API v1.0.0, Component v3.1.3)
MCA rmaps: mindist (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA rmaps: ppr (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA rmaps: rank_file (MCA v2.1.0, API v2.0.0, Component
v3.1.3)
MCA rmaps: resilient (MCA v2.1.0, API v2.0.0, Component
v3.1.3)
MCA rmaps: round_robin (MCA v2.1.0, API v2.0.0, Component
v3.1.3)
MCA rmaps: seq (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA rml: ofi (MCA v2.1.0, API v3.0.0, Component v3.1.3)
MCA rml: oob (MCA v2.1.0, API v3.0.0, Component v3.1.3)
MCA routed: binomial (MCA v2.1.0, API v3.0.0, Component v3.1.3)
MCA routed: debruijn (MCA v2.1.0, API v3.0.0, Component v3.1.3)
MCA routed: direct (MCA v2.1.0, API v3.0.0, Component v3.1.3)
MCA routed: radix (MCA v2.1.0, API v3.0.0, Component v3.1.3)
MCA rtc: hwloc (MCA v2.1.0, API v1.0.0, Component v3.1.3)
MCA schizo: flux (MCA v2.1.0, API v1.0.0, Component v3.1.3)
MCA schizo: ompi (MCA v2.1.0, API v1.0.0, Component v3.1.3)
MCA schizo: orte (MCA v2.1.0, API v1.0.0, Component v3.1.3)
MCA schizo: slurm (MCA v2.1.0, API v1.0.0, Component v3.1.3)
MCA state: app (MCA v2.1.0, API v1.0.0, Component v3.1.3)
MCA state: dvm (MCA v2.1.0, API v1.0.0, Component v3.1.3)
MCA state: hnp (MCA v2.1.0, API v1.0.0, Component v3.1.3)
MCA state: novm (MCA v2.1.0, API v1.0.0, Component v3.1.3)
MCA state: orted (MCA v2.1.0, API v1.0.0, Component v3.1.3)
MCA state: tool (MCA v2.1.0, API v1.0.0, Component v3.1.3)
MCA bml: r2 (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA coll: basic (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA coll: inter (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA coll: libnbc (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA coll: monitoring (MCA v2.1.0, API v2.0.0, Component
v3.1.3)
MCA coll: self (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA coll: sm (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA coll: spacc (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA coll: sync (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA coll: tuned (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA fbtl: posix (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA fcoll: dynamic (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA fcoll: dynamic_gen2 (MCA v2.1.0, API v2.0.0, Component
v3.1.3)
MCA fcoll: individual (MCA v2.1.0, API v2.0.0, Component
v3.1.3)
MCA fcoll: static (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA fcoll: two_phase (MCA v2.1.0, API v2.0.0, Component
v3.1.3)
MCA fs: ufs (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA io: romio314 (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA io: ompio (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA mtl: ofi (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA mtl: psm (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA mtl: psm2 (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA osc: ucx (MCA v2.1.0, API v3.0.0, Component v3.1.3)
MCA osc: monitoring (MCA v2.1.0, API v3.0.0, Component
v3.1.3)
MCA osc: pt2pt (MCA v2.1.0, API v3.0.0, Component v3.1.3)
MCA osc: rdma (MCA v2.1.0, API v3.0.0, Component v3.1.3)
MCA osc: sm (MCA v2.1.0, API v3.0.0, Component v3.1.3)
MCA pml: v (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA pml: cm (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA pml: monitoring (MCA v2.1.0, API v2.0.0, Component
v3.1.3)
MCA pml: ob1 (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA pml: ucx (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA rte: orte (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA sharedfp: individual (MCA v2.1.0, API v2.0.0, Component
v3.1.3)
MCA sharedfp: sm (MCA v2.1.0, API v2.0.0, Component v3.1.3)
MCA sharedfp: lockedfile (MCA v2.1.0, API v2.0.0, Component
v3.1.3)
MCA topo: basic (MCA v2.1.0, API v2.2.0, Component v3.1.3)
MCA topo: treematch (MCA v2.1.0, API v2.2.0, Component
v3.1.3)
MCA vprotocol: pessimist (MCA v2.1.0, API v2.0.0, Component
v3.1.3)
谢谢你！