OpenAI API max_tokens 计数

Question

我遇到了一个非常奇怪的问题。我正在运行此curl请求，向

hi

发送一条简单的

gpt-4

消息。模型的最大上下文长度为 8192。

hi

应为 1 个令牌（您可以在此处查看 - https://platform.openai.com/tokenizer）。这意味着我可以将

max_tokens

设置为

8192 - 1 = 8191

。

但是，OpenAI 的 API 计算出我的消息需要 8 个令牌。为什么？

curl https://api.openai.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-bla-bla-bla" \
  -d '{
    "model": "gpt-4",
    "max_tokens": 8191,
    "messages": [
      {
        "role": "user",
        "content": "hi"
      }
    ]
  }'
{
  "error": {
    "message": "This model's maximum context length is 8192 tokens. However, you requested 8199 tokens (8 in the messages, 8191 in the completion). Please reduce the length of the messages or completion.",
    "type": "invalid_request_error",
    "param": "messages",
    "code": "context_length_exceeded"
  }
}

Answer 1

刚刚在这里找到答案，疯狂： https://community.openai.com/t/what-is-the-reason-for-adding-total-7-tokens/337002