PMPI和otf2：在CPP程序中链接C代码

Question

我有一个由wrap.py生成的CPP程序。 wrap.py用于为MPI程序生成包装器。它将任何正常的MPI呼叫重定向到PMPI呼叫以进行拦截，以便例如进行拦截。绩效分析。请下载生成的代码here。我使用otf2来跟踪MPI程序。

解释代码：

// test4.cpp
__attribute__((constructor)) void init(void)
{
  if(!is_init)
  {
    archive = OTF2_Archive_Open( "./",
                                 "ArchiveTest",
                                 OTF2_FILEMODE_WRITE,
                                 1024 * 1024 /* event chunk size */,
                                 4 * 1024 * 1024 /* def chunk size */,
                                 OTF2_SUBSTRATE_POSIX,
                                 OTF2_COMPRESSION_NONE );
    is_init = true;
  }
}

__attribute__((destructor))  void fini(void)
{
  if(is_init)
  {
    OTF2_Archive_Close( archive );
    is_init = false;
  }
}

我打算将代码编译成.so文件。因此，当它被导入时，将调用constructor;当.so脱离时，destructor被召唤。

根据otf2here的官方文档，我编译程序：

mpic++ -fpic -c `otf2-config --cflags` -o test4.o test4.cpp
mpic++ -shared -o libtest4.so `otf2-config --ldflags` `otf2-config --libs` test4.o

如果扩展上面的命令行，你会得到：

mpic++ -fpic -c -I/usr/include -o test4.o test4.cpp
mpic++ -shared -o libtest4.so -L/usr/lib -lotf2 -lm test4.o

截获的MPI计划来自here。

截取：

$ mpirun -n 2 -x LD_PRELOAD=./libtest4.so ./send_recv
./send_recv: symbol lookup error: ./libtest4.so: undefined symbol: OTF2_Archive_Open
./send_recv: symbol lookup error: ./libtest4.so: undefined symbol: OTF2_Archive_Open
-------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code.. Per user-direction, the job has been aborted.
-------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[20246,1],0]
  Exit code:    127
--------------------------------------------------------------------------

所以看起来混合C和CPP会导致问题。链接器无法正确生成C函数的符号，即OTF2_Archive_Open和OTF2_Archive_Close。

我添加了2个声明来告诉链接器那些是C函数（下载修改后的programhere）：

_EXTERN_C_ OTF2_Archive* OTF2_Archive_Open ( const char *  archivePath,
const char *  archiveName,
const OTF2_FileMode   fileMode,
const uint64_t  chunkSizeEvents,
const uint64_t  chunkSizeDefs,
const OTF2_FileSubstrate  fileSubstrate,
const OTF2_Compression  compression
);
_EXTERN_C_ OTF2_ErrorCode OTF2_Archive_Close ( OTF2_Archive *  archive );

但上面的问题仍然存在。和建议？

UPDATE1：OTF2提供.a文件，而不是.so文件。

$ nm /usr/lib/libotf2.a| grep -i OTF2_Archive_Open
                 U otf2_archive_open
0000000000000000 T OTF2_Archive_Open
                 U otf2_archive_open_def_files
00000000000032e0 T OTF2_Archive_OpenDefFiles
                 U otf2_archive_open_evt_files
00000000000030e0 T OTF2_Archive_OpenEvtFiles
                 U otf2_archive_open_snap_files
00000000000034e0 T OTF2_Archive_OpenSnapFiles
                 U OTF2_Archive_Open
0000000000001180 T otf2_archive_open
0000000000005a40 T otf2_archive_open_def_files
                 U OTF2_Archive_OpenDefFiles
0000000000005880 T otf2_archive_open_evt_files
                 U OTF2_Archive_OpenEvtFiles
0000000000005c00 T otf2_archive_open_snap_files
                 U OTF2_Archive_OpenSnapFiles


$ ldd ./libtest4.so
    linux-vdso.so.1 =>  (0x00007ffe3a6ce000)
    libmpi_cxx.so.1 => /usr/lib/libmpi_cxx.so.1 (0x00007f4757d67000)
    libmpi.so.12 => /usr/lib/libmpi.so.12 (0x00007f4757a91000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f475770e000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f47574f8000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f475712e000)
    libibverbs.so.1 => /usr/lib/libibverbs.so.1 (0x00007f4756f1e000)
    libopen-rte.so.12 => /usr/lib/libopen-rte.so.12 (0x00007f4756ca4000)
    libopen-pal.so.13 => /usr/lib/libopen-pal.so.13 (0x00007f4756a07000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f47567e9000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f47564e0000)
    /lib64/ld-linux-x86-64.so.2 (0x00005620bef03000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f47562dc000)
    libhwloc.so.5 => /usr/lib/x86_64-linux-gnu/libhwloc.so.5 (0x00007f47560a1000)
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f4755e99000)
    libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f4755c96000)
    libnuma.so.1 => /usr/lib/x86_64-linux-gnu/libnuma.so.1 (0x00007f4755a8a000)
    libltdl.so.7 => /usr/lib/x86_64-linux-gnu/libltdl.so.7 (0x00007f4755880000)



$ nm ./libtest4.so | grep -i OTF2_Archive_Open
                 U OTF2_Archive_Open

奇怪的是，我没有在libotf2.a的输出中看到任何ldd。但是如果你从他们的网站上试用otf2 mpi writer的标准例子，那就可以了。而且，对于otf2 mpi writer的标准示例，ldd的输出也不包含libotf2.a。

你可以找到例子here。

Answer 1

关联的顺序。您必须在链接的库前放置自己的库，例如

mpic++ -shared test4.o -o libtest4.so `otf2-config --ldflags` `otf2-config --libs`

链接器从左到右解析未知符号。有关更多详细信息，请参阅this answer。如果otf2.a不是用-fPIC构建的话，这可能仍然无效。我建议用--enable-shared配置otf2并使用.so代替。