如何在 pytorch 中为 ppc64le 架构安装 torchmeta?

问题描述 投票:0回答:1

我尝试在 ppc64le 架构中使用 torchmeta。不幸的是,安装起来并不容易,因为 ppc64le 需要特殊的二进制文件才能工作。

我最终按照这些说明成功获得了 pytorch 和 torchvision 的正确二进制文件(在正确的 ibm 通道前面加上 conda 二进制文件,并且还安装了所有必需的文件):

conda config --prepend channels https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/ conda create -n my_new_env python=3.7 powerai=1.7.0 conda activate my_new_env
之后我继续安装正确版本的torchmeta,这是

1.3.1

,因为ppc64le只有pytorch
1.3.1
和torchvision
0.4.2
。所以我就这么做了:

pip install torchmeta==1.3.1
但现在我有一个新错误,它无法找到与我想做的事情兼容的 h5py 的正确版本。错误消息太大,无法粘贴,但我将粘贴我希望其中有用的部分:

(my_new_env) [miranda9@hal-login ~]$ pip install torchmeta==1.3.1 Collecting torchmeta==1.3.1 Using cached torchmeta-1.3.1-py3-none-any.whl (144 kB) Requirement already satisfied: requests in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from torchmeta==1.3.1) (2.22.0) Requirement already satisfied: torchvision<0.6.0,>=0.4.0 in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from torchmeta==1.3.1) (0.4.2) Requirement already satisfied: torch<1.5.0,>=1.3.0 in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from torchmeta==1.3.1) (1.3.1) Processing ./.cache/pip/wheels/87/f5/ad/9f04a48453875e8054c19f9fe3f50cbbe0c09b956835555019/Pillow-6.2.2-cp37-cp37m-linux_ppc64le.whl Requirement already satisfied: numpy>=1.14.0 in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from torchmeta==1.3.1) (1.17.4) Requirement already satisfied: tqdm>=4.0.0 in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from torchmeta==1.3.1) (4.36.1) Collecting h5py~=2.9.0 Using cached h5py-2.9.0.tar.gz (287 kB) Requirement already satisfied: chardet<3.1.0,>=3.0.2 in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from requests->torchmeta==1.3.1) (3.0.4) Requirement already satisfied: certifi>=2017.4.17 in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from requests->torchmeta==1.3.1) (2020.6.20) Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from requests->torchmeta==1.3.1) (1.25.10) Requirement already satisfied: idna<2.9,>=2.5 in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from requests->torchmeta==1.3.1) (2.8) Requirement already satisfied: six in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from torchvision<0.6.0,>=0.4.0->torchmeta==1.3.1) (1.13.0) Building wheels for collected packages: h5py Building wheel for h5py (setup.py) ... error ERROR: Command errored out with exit status 1: command: /home/miranda9/.conda/envs/my_new_env/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-bpmeop26/h5py/setup.py'"'"'; __file__='"'"'/tmp/pip-install-bpmeop26/h5py/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-ccg1oj0n cwd: /tmp/pip-install-bpmeop26/h5py/ Complete output (1321 lines): running bdist_wheel running build running build_py creating build creating build/lib.linux-ppc64le-3.7 creating build/lib.linux-ppc64le-3.7/h5py copying h5py/__init__.py -> build/lib.linux-ppc64le-3.7/h5py copying h5py/h5py_warnings.py -> build/lib.linux-ppc64le-3.7/h5py copying h5py/highlevel.py -> build/lib.linux-ppc64le-3.7/h5py copying h5py/ipy_completer.py -> build/lib.linux-ppc64le-3.7/h5py copying h5py/version.py -> build/lib.linux-ppc64le-3.7/h5py creating build/lib.linux-ppc64le-3.7/h5py/_hl copying h5py/_hl/__init__.py -> build/lib.linux-ppc64le-3.7/h5py/_hl copying h5py/_hl/attrs.py -> build/lib.linux-ppc64le-3.7/h5py/_hl copying h5py/_hl/base.py -> build/lib.linux-ppc64le-3.7/h5py/_hl copying h5py/_hl/compat.py -> build/lib.linux-ppc64le-3.7/h5py/_hl copying h5py/_hl/dataset.py -> build/lib.linux-ppc64le-3.7/h5py/_hl copying h5py/_hl/datatype.py -> build/lib.linux-ppc64le-3.7/h5py/_hl copying h5py/_hl/dims.py -> build/lib.linux-ppc64le-3.7/h5py/_hl copying h5py/_hl/files.py -> build/lib.linux-ppc64le-3.7/h5py/_hl copying h5py/_hl/filters.py -> build/lib.linux-ppc64le-3.7/h5py/_hl copying h5py/_hl/group.py -> build/lib.linux-ppc64le-3.7/h5py/_hl copying h5py/_hl/selections.py -> build/lib.linux-ppc64le-3.7/h5py/_hl copying h5py/_hl/selections2.py -> build/lib.linux-ppc64le-3.7/h5py/_hl copying h5py/_hl/vds.py -> build/lib.linux-ppc64le-3.7/h5py/_hl creating build/lib.linux-ppc64le-3.7/h5py/tests copying h5py/tests/__init__.py -> build/lib.linux-ppc64le-3.7/h5py/tests copying h5py/tests/common.py -> build/lib.linux-ppc64le-3.7/h5py/tests creating build/lib.linux-ppc64le-3.7/h5py/tests/old copying h5py/tests/old/__init__.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old copying h5py/tests/old/test_attrs.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old copying h5py/tests/old/test_attrs_data.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old copying h5py/tests/old/test_base.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old copying h5py/tests/old/test_dataset.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old copying h5py/tests/old/test_datatype.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old copying h5py/tests/old/test_dimension_scales.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old copying h5py/tests/old/test_file.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old copying h5py/tests/old/test_file_image.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old copying h5py/tests/old/test_group.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old copying h5py/tests/old/test_h5.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old copying h5py/tests/old/test_h5d_direct_chunk_write.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old copying h5py/tests/old/test_h5f.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old copying h5py/tests/old/test_h5p.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old copying h5py/tests/old/test_h5t.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old copying h5py/tests/old/test_objects.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old copying h5py/tests/old/test_selections.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old copying h5py/tests/old/test_slicing.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old creating build/lib.linux-ppc64le-3.7/h5py/tests/hl copying h5py/tests/hl/__init__.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl copying h5py/tests/hl/test_attribute_create.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl copying h5py/tests/hl/test_dataset_getitem.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl copying h5py/tests/hl/test_dataset_swmr.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl copying h5py/tests/hl/test_datatype.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl copying h5py/tests/hl/test_deprecation.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl copying h5py/tests/hl/test_dims_dimensionproxy.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl copying h5py/tests/hl/test_file.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl copying h5py/tests/hl/test_filters.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl copying h5py/tests/hl/test_threads.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl creating build/lib.linux-ppc64le-3.7/h5py/tests/hl/test_vds copying h5py/tests/hl/test_vds/__init__.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl/test_vds copying h5py/tests/hl/test_vds/test_highlevel_vds.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl/test_vds copying h5py/tests/hl/test_vds/test_lowlevel_vds.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl/test_vds copying h5py/tests/hl/test_vds/test_virtual_source.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl/test_vds running build_ext Autodetected HDF5 1.10.2 ******************************************************************************** Summary of the h5py configuration Path to HDF5: None HDF5 Version: '1.10.2' MPI Enabled: False Rebuild Required: True ******************************************************************************** Executing api_gen rebuild of defs Executing cythonize() [ 1/22] Cythonizing /tmp/pip-install-bpmeop26/h5py/h5py/_conv.pyx /tmp/pip-install-bpmeop26/h5py/.eggs/Cython-0.29.21-py3.7.egg/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /tmp/pip-install-bpmeop26/h5py/h5py/_conv.pxd tree = Parsing.p_module(s, pxd, full_module_name) ... /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp] #warning "Using deprecated NumPy API, disable it with " \ ^ In file included from /tmp/pip-install-bpmeop26/h5py/h5py/defs.c:654:0: /tmp/pip-install-bpmeop26/h5py/h5py/api_compat.h:27:18: fatal error: hdf5.h: No such file or directory #include "hdf5.h" ^ compilation terminated. error: command 'gcc' failed with exit status 1 ---------------------------------------- ERROR: Failed building wheel for h5py Running setup.py clean for h5py Failed to build h5py DEPRECATION: Could not build wheels for h5py which do not use PEP 517. pip will fall back to legacy 'setup.py install' for these. pip 21.0 will remove support for this functionality. A possible replacement is to fix the wheel build issue reported above. You can find discussion regarding this at https://github.com/pypa/pip/issues/8368. Installing collected packages: Pillow, h5py, torchmeta Attempting uninstall: Pillow Found existing installation: Pillow 7.1.2 Uninstalling Pillow-7.1.2: Successfully uninstalled Pillow-7.1.2 Attempting uninstall: h5py Found existing installation: h5py 2.8.0 Uninstalling h5py-2.8.0: Successfully uninstalled h5py-2.8.0 Running setup.py install for h5py ... error ERROR: Command errored out with exit status 1: command: /home/miranda9/.conda/envs/my_new_env/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-bpmeop26/h5py/setup.py'"'"'; __file__='"'"'/tmp/pip-install-bpmeop26/h5py/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-hlwpfooj/install-record.txt --single-version-externally-managed --compile --install-headers /home/miranda9/.conda/envs/my_new_env/include/python3.7m/h5py ... copying h5py/tests/hl/test_vds/test_lowlevel_vds.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl/test_vds copying h5py/tests/hl/test_vds/test_virtual_source.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl/test_vds running build_ext Autodetected HDF5 1.10.2 ******************************************************************************** Summary of the h5py configuration Path to HDF5: None HDF5 Version: '1.10.2' MPI Enabled: False Rebuild Required: True ******************************************************************************** Executing cythonize() [ 1/22] Cythonizing /tmp/pip-install-bpmeop26/h5py/h5py/_conv.pyx /tmp/pip-install-bpmeop26/h5py/.eggs/Cython-0.29.21-py3.7.egg/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /tmp/pip-install-bpmeop26/h5py/h5py/_conv.pxd ... warning: h5py/api_types_hdf5.pxd:730:6: 'H5Z_ERROR_EDC' redeclared warning: h5py/api_types_hdf5.pxd:731:6: 'H5Z_DISABLE_EDC' redeclared warning: h5py/api_types_hdf5.pxd:732:6: 'H5Z_ENABLE_EDC' redeclared warning: h5py/api_types_hdf5.pxd:733:6: 'H5Z_NO_EDC' redeclared building 'h5py.defs' extension creating build/temp.linux-ppc64le-3.7 creating build/temp.linux-ppc64le-3.7/tmp creating build/temp.linux-ppc64le-3.7/tmp/pip-install-bpmeop26 creating build/temp.linux-ppc64le-3.7/tmp/pip-install-bpmeop26/h5py creating build/temp.linux-ppc64le-3.7/tmp/pip-install-bpmeop26/h5py/h5py gcc -pthread -B /home/miranda9/.conda/envs/my_new_env/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DH5_USE_16_API -I./h5py -I/tmp/pip-install-bpmeop26/h5py/lzf -I/opt/local/include -I/usr/local/include -I/home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/numpy/core/include -I/home/miranda9/.conda/envs/my_new_env/include/python3.7m -c /tmp/pip-install-bpmeop26/h5py/h5py/defs.c -o build/temp.linux-ppc64le-3.7/tmp/pip-install-bpmeop26/h5py/h5py/defs.o In file included from /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1830:0, from /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:12, from /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4, from /tmp/pip-install-bpmeop26/h5py/h5py/api_compat.h:26, from /tmp/pip-install-bpmeop26/h5py/h5py/defs.c:654: /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp] #warning "Using deprecated NumPy API, disable it with " \ ^ In file included from /tmp/pip-install-bpmeop26/h5py/h5py/defs.c:654:0: /tmp/pip-install-bpmeop26/h5py/h5py/api_compat.h:27:18: fatal error: hdf5.h: No such file or directory #include "hdf5.h" ^ compilation terminated. error: command 'gcc' failed with exit status 1 ---------------------------------------- Rolling back uninstall of h5py Moving to /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/h5py from /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/~5py Moving to /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/h5py-2.8.0-py3.7.egg-info from /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/~5py-2.8.0-py3.7.egg-info ERROR: Command errored out with exit status 1: /home/miranda9/.conda/envs/my_new_env/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-bpmeop26/h5py/setup.py'"'"'; __file__='"'"'/tmp/pip-install-bpmeop26/h5py/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-hlwpfooj/install-record.txt --single-version-externally-managed --compile --install-headers /home/miranda9/.conda/envs/my_new_env/include/python3.7m/h5py Check the logs for full command output.
有人知道如何在 ppc64le 中成功安装工作的 torchmeta 版本(使用 wmcle 1.7.0)吗?


相关:

  • torchmeta 的 gitissue:

    https://github.com/tristandeleu/pytorch-meta/issues/95

  • 用于 torchmeta 支持的 IBM gitissue:

    https://github.com/IBM/powerai/issues/269

  • 用于 torchmeta 的 h5py gitissue:

    https://github.com/h5py/h5py/issues/1678

  • IBM h5py 对 torchmeta 的支持:

    https://github.com/IBM/powerai/issues/270

python pytorch conda ppc64le
1个回答
3
投票
因为 h5py 的 powerpc 没有轮子,所以您要从源代码(从 tarball)安装 h5py。这需要 Python 和 h5py 开发标头都可用,请参阅

https://docs.h5py.org/en/stable/build.html#source-installation

从 conda 安装 h5py 或安装所需的构建依赖项。

© www.soinside.com 2019 - 2024. All rights reserved.