使用 Pytest 并行化 Cython 代码进行基准测试会导致致命的 Python 错误

Question

我有以下测试：

import array
def test_parallel_cython_clip(benchmark: Any) -> None:
    benchmark.pedantic(
        math_cython.parallel_cython_clip_vector,
        args=(array.array("f", [0.0]),-1.0,1.0,array.array("f", [0.0])),
        rounds=1,
        iterations=1
    )

我通过以下方式执行：

pytest --benchmark-columns=min,max,mean,stddev --benchmark-sort=mean benchmark.py -vv -k test_parallel_cython_clip

这会导致错误：

benchmark.py::test_parallel_cython_clip Fatal Python error: Aborted

此外还提供以下信息：

Extension modules: mkl._mklinit, mkl._py_mkl_service, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, math_cython.cython_computations (total: 16)

可编辑安装到 conda 环境中的包文件夹

math_cython

包含一个

.py

模块，其中包含以下条目：

def parallel_cython_clip_vector(
    vector_in: array.array,
    min_value: float,
    max_value: float,
    vector_out: array.array,
) -> None:
    _parallel_cython_clip_vector(vector_in, min_value, max_value, vector_out)

和一个

.pyx

文件，其

_parallel_cython_clip_vector

函数定义如下：

@cython.boundscheck(False)  # Deactivate bounds checking (which is possible in Python, but not in C)
@cython.wraparound(False)  # Deactivate negative indexing (which is possible in Python, but not in C)
def _parallel_cython_clip_vector(
    float[:] vector_in,
    float min_value,
    float max_value,
    float[:] vector_out,
):
    cdef signed int idx = 0
    <this part I commented for the moment>

操作系统信息：

macOS 14.3.1 
Darwin 23.3.0

已安装的Python库：

cython 3.0.6
pytest 8.0.0
pytest-benchmark 4.0.0
python 3.11.7
numpy 1.26.3

我在这里看到了类似的问题，但我认为他们没有解决我的问题：

• 使用 Cython 实现并行
• Cython prange 失败并出现致命 Python 错误：PyThreadState_Get：没有当前线程
• Cython 使用内存视图崩溃了

任何解决致命Python错误的帮助都将非常感激。

编辑：在 VSCode 中运行调试器我得到了有关错误的更多信息：

OMP: Error #15: Initializing libomp.dylib, but found libiomp5.dylib already initialized.
OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see http://openmp.llvm.org/
Fatal Python error: Aborted

Answer 1

我的问题的解决方案是：

conda install nomkl

MacOS 上需要这样做的原因在这里有很好的解释： https://stackoverflow.com/a/58869103/9698518

我想说的是，Conda 的 Intel MKL 优化与 macOS 自己的加速框架（也随 OpenMP 一起提供）之间存在冲突。