无法从PyPI安装tiktoken

Question

我正在致力于一些 OpenAI 的 API 集成。我在安装 tiktoken（一种与 OpenAI 模型一起使用的快速 BPE 标记器）时遇到问题。

当我尝试在我的 MacBook 机器上运行时：

pip3 install tiktoken -v

它抛出以下错误（使用 pip 22.3.1，错误因太长而被截断）：

  *********************************************************************************
  Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?
  Perhaps try: xcode-select --install
  *********************************************************************************
  error: command '/usr/bin/clang' failed with exit code 1
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /Users/gianlucatrentin/gptenv/bin/python3 -u -c '
  exec(compile('"'"''"'"''"'"'
  # This is <pip-setuptools-caller> -- a caller that pip uses to run setup.py
  #
  # - It imports setuptools before invoking setup.py, to enable projects that directly
  #   import from `distutils.core` to work with newer packaging standards.
  # - It provides a clear error message when setuptools is not installed.
  # - It sets `sys.argv[0]` to the underlying `setup.py`, when invoking `setup.py` so
  #   setuptools doesn'"'"'t think the script is `-c`. This avoids the following warning:
  #     manifest_maker: standard file '"'"'-c'"'"' not found".
  # - It generates a shim setup.py, for handling setup.cfg-only projects.
  import os, sys, tokenize
  
  try:
      import setuptools
  except ImportError as error:
      print(
          "ERROR: Can not execute `setup.py` since setuptools is not available in "
          "the build environment.",
          file=sys.stderr,
      )
      sys.exit(1)
  
  __file__ = %r
  sys.argv[0] = __file__
  
  if os.path.exists(__file__):
      filename = __file__
      with tokenize.open(__file__) as f:
          setup_py_code = f.read()
  else:
      filename = "<auto-generated setuptools caller>"
      setup_py_code = "from setuptools import setup; setup()"
  
  exec(compile(setup_py_code, filename, "exec"))
  '"'"''"'"''"'"' % ('"'"'/private/var/folders/sc/gl1552412gb0lg7_x33zrgj00000gn/T/pip-install-l7tec_bv/lxml_b78f1cdebea441d2b840e6957fabe83a/setup.py'"'"',), "<pip-setuptools-caller>", "exec"))' bdist_wheel -d /private/var/folders/sc/gl1552412gb0lg7_x33zrgj00000gn/T/pip-wheel-sd9x1ng7
  cwd: /private/var/folders/sc/gl1552412gb0lg7_x33zrgj00000gn/T/pip-install-l7tec_bv/lxml_b78f1cdebea441d2b840e6957fabe83a/
  Building wheel for lxml (setup.py) ... error
  ERROR: Failed building wheel for lxml
  Running setup.py clean for lxml
  Running command python setup.py clean
  Building lxml version 4.9.2.
  Building without Cython.
  Building against libxml2 2.9.13 and libxslt 1.1.35
  running clean
  removing 'build/temp.macosx-10.9-universal2-cpython-310' (and everything under it)
  removing 'build/lib.macosx-10.9-universal2-cpython-310' (and everything under it)
  'build/bdist.macosx-10.9-universal2' does not exist -- can't clean it
  'build/scripts-3.10' does not exist -- can't clean it
  removing 'build'
 ...
  × Running setup.py install for pycryptodomex did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /Users/gianlucatrentin/gptenv/bin/python3 -u -c '
  exec(compile('"'"''"'"''"'"'
  # This is <pip-setuptools-caller> -- a caller that pip uses to run setup.py
  #
  # - It imports setuptools before invoking setup.py, to enable projects that directly
  #   import from `distutils.core` to work with newer packaging standards.
  # - It provides a clear error message when setuptools is not installed.
  # - It sets `sys.argv[0]` to the underlying `setup.py`, when invoking `setup.py` so
  #   setuptools doesn'"'"'t think the script is `-c`. This avoids the following warning:
  #     manifest_maker: standard file '"'"'-c'"'"' not found".
  # - It generates a shim setup.py, for handling setup.cfg-only projects.
  import os, sys, tokenize
  
  try:
      import setuptools
  except ImportError as error:
      print(
          "ERROR: Can not execute `setup.py` since setuptools is not available in "
          "the build environment.",
          file=sys.stderr,
      )
      sys.exit(1)
  
  __file__ = %r
  sys.argv[0] = __file__
  
  if os.path.exists(__file__):
      filename = __file__
      with tokenize.open(__file__) as f:
          setup_py_code = f.read()
  else:
      filename = "<auto-generated setuptools caller>"
      setup_py_code = "from setuptools import setup; setup()"
  
  exec(compile(setup_py_code, filename, "exec"))
  '"'"''"'"''"'"' % ('"'"'/private/var/folders/sc/gl1552412gb0lg7_x33zrgj00000gn/T/pip-install-l7tec_bv/pycryptodomex_7192f9e9e3b34b15b1960598d9d7eb02/setup.py'"'"',), "<pip-setuptools-caller>", "exec"))' install --record /private/var/folders/sc/gl1552412gb0lg7_x33zrgj00000gn/T/pip-record-n_aypwf1/install-record.txt --single-version-externally-managed --compile --install-headers /Users/gianlucatrentin/gptenv/include/site/python3.10/pycryptodomex
  cwd: /private/var/folders/sc/gl1552412gb0lg7_x33zrgj00000gn/T/pip-install-l7tec_bv/pycryptodomex_7192f9e9e3b34b15b1960598d9d7eb02/
  Running setup.py install for pycryptodomex ... error
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> pycryptodomex

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

我尝试升级所有提及的库，但无法使其工作。 您对如何解决此安装问题有什么建议吗？

Answer 1

我在为我的语言模型安装 tiktoken 时遇到了这个问题。我通过以下方式安装成功了

首先升级你的 pip 和 setuptools。

然后通过

安装setuptools_rust

pip install setuptools_rust

然后只需安装libxml2、libxlst即可

pkg install libxml2, libxslt

如果仍然不起作用，如果遇到“找不到 Rust 编译器”，请尝试通过

安装 Rust

pkg install rust

完成所有这些步骤后，就应该安装了。

希望对您有帮助...

Answer 2

您可以执行以下两项操作之一：

1. 按照以下说明安装 Rust：https://rustup.rs/

或者（如果你无法安装 Rust）

2. 从wheel文件安装tiktoken，而不是自己构建wheel。