我无法使用pip在python上安装tiktoken

Question

我正在尝试运行一些Python代码，其中包括一些tiktoken调用。

我尝试为 python 安装 tiktoken 但不能。我正在尝试使用 pip 命令安装，但收到有关 rust 的错误。我想知道是否应该在 python 或 pip 之外安装一些 rust“东西”。

Collecting tiktoken
  Using cached tiktoken-0.3.3.tar.gz (25 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: requests>=2.26.0 in g:\program files\python\lib\site-packages (from tiktoken) (2.28.2)
Collecting regex>=2022.1.18
  Using cached regex-2023.3.23-cp38-cp38-win32.whl (256 kB)
Requirement already satisfied: idna<4,>=2.5 in g:\program files\python\lib\site-packages (from requests>=2.26.0->tiktoken) (2.9)
Requirement already satisfied: certifi>=2017.4.17 in g:\program files\python\lib\site-packages (from requests>=2.26.0->tiktoken) (2020.4.5.1)
Requirement already satisfied: charset-normalizer<4,>=2 in g:\program files\python\lib\site-packages (from requests>=2.26.0->tiktoken) (3.1.0)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in g:\program files\python\lib\site-packages (from requests>=2.26.0->tiktoken) (1.26.15)
Building wheels for collected packages: tiktoken
  Building wheel for tiktoken (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for tiktoken (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [37 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build\lib.win32-cpython-38
      creating build\lib.win32-cpython-38\tiktoken
      copying tiktoken\core.py -> build\lib.win32-cpython-38\tiktoken
      copying tiktoken\load.py -> build\lib.win32-cpython-38\tiktoken
      copying tiktoken\model.py -> build\lib.win32-cpython-38\tiktoken
      copying tiktoken\registry.py -> build\lib.win32-cpython-38\tiktoken
      copying tiktoken\__init__.py -> build\lib.win32-cpython-38\tiktoken
      creating build\lib.win32-cpython-38\tiktoken_ext
      copying tiktoken_ext\openai_public.py -> build\lib.win32-cpython-38\tiktoken_ext
      running egg_info
      writing tiktoken.egg-info\PKG-INFO
      writing dependency_links to tiktoken.egg-info\dependency_links.txt
      writing requirements to tiktoken.egg-info\requires.txt
      writing top-level names to tiktoken.egg-info\top_level.txt
      reading manifest file 'tiktoken.egg-info\SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      warning: no files found matching 'Makefile'
      adding license file 'LICENSE'
      writing manifest file 'tiktoken.egg-info\SOURCES.txt'
      copying tiktoken\py.typed -> build\lib.win32-cpython-38\tiktoken
      running build_ext
      running build_rust
      error: can't find Rust compiler
      
      If you are using an outdated pip version, it is possible a prebuilt wheel is available for this package but pip is not able to install from it. Installing from the wheel would avoid the 
need for a Rust compiler.
     
      To update pip, run:
     
          pip install --upgrade pip
     
      and then retry package installation.
     
      If you did intend to build this package from source, try installing a Rust compiler from your system package manager and ensure it is on the PATH during installation. Alternatively, rustup (available at https://rustup.rs) is the recommended way to download and update the Rust compiler toolchain.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for tiktoken
Failed to build tiktoken
ERROR: Could not build wheels for tiktoken, which is required to install pyproject.toml-based projects

我尝试过“pip install tiktoken”。

Answer 1

Pip 正在尝试从源代码构建 tiktoken 库，而您缺少 Rust 编译器。您可以在系统上安装 Rust 编译器，或者从轮子安装 tiktoken，而不是从源代码构建它。

请在 tiktoken GitHub 存储库中查看此问题以获得更多帮助

Answer 2

Pip 需要从源代码安装 tiktoken，这需要 Rust 编译器。

我找不到可以安装的轮子，所以这里是安装 Rust 编译器的方法：

1. 通过 rustup 下载 Rust（使用默认安装）：

curl --proto '=https' --tlsv1.3 https://sh.rustup.rs -sSf |嘘

2. 重新启动终端并获取货物的 bin 目录：

。 "$HOME/.cargo/env" # 对于 sh/bash/zsh/ash/dash/pdksh

source "$HOME/.cargo/env.fish" # 对于鱼

3. 通过运行验证安装：

rustc --版本

4. 安装编译器：

sudo apt install build-essential