是否可以从非交互式后端调用 Google 的 Imagen API？

Question

我的目标是从非交互式后端以 QnA 模式使用 Imagen。 文档 (https://console.cloud.google.com/vertex-ai/publishers/google/model-garden/imagetext-vqa?project=gdg-demos&cloudshell=true) 使用

gcloud auth print-access-token 填写不记名令牌

命令。如果我在云 shell 中执行它，我会得到一个令牌，但它在非交互式后端中无法使用。

    base64_string = base64_bytes.decode(ENCODING)

    VQA_PROMPT = "Describe the content of the image in great detail"

    payload = {
      "instances": [
        {
          "prompt": VQA_PROMPT,
          "image": {
              "bytesBase64Encoded": base64_string
          }
        }
      ],
      "parameters": parameters
    }

    url = "https://us-central1-aiplatform.googleapis.com/v1/projects/gdg-demos/locations/us-central1/publishers/google/models/imagetext:predict"
    headers = {
        "Authorization": "Bearer {}".format(bearer_token),
        "Accept": "application/json; charset=utf-8",
    }
    json_data = requests.post(url, headers=headers, json=payload)

我收到 401 HTTP 状态代码响应：

b'{
  "error": {
    "code": 401,
    "message": "Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.",
    "status": "UNAUTHENTICATED",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.ErrorInfo",
        "reason": "ACCESS_TOKEN_TYPE_UNSUPPORTED",
        "metadata": {
          "service": "aiplatform.googleapis.com",
          "method": "google.cloud.aiplatform.v1.PredictionService.Predict"
        }
      }
    ]
  }
}'

---

我尝试过https://saturncloud.io/blog/authenticate-to-google-container-service-with-script-noninteractive-gcloud-auth-login/

1. 向 GCS 进行身份验证：

   gcloud auth login --brief --quiet

2. 检索刷新令牌：

   REFRESH_TOKEN=$(gcloud auth print-access-token)

3. 激活刷新令牌：

   gcloud auth activate-refresh-token $REFRESH_TOKEN

我用我正在修改的 JupyterLab 打开了一个终端。我能够激活刷新令牌，并在第三步后得到

Activated refresh token credentials: [***]

。然后我尝试使用该令牌作为 Bearer 令牌，但我收到了带有

Forbidden

的 403 HTTP 状态代码。 如果我在该终端中执行常规（非简短且非安静）

gcloud auth print-access-token

并也尝试了该令牌，但也得到了 403，则相同。

Answer 1

感谢 Google 的 Anish Nangia 指出我正在查看错误的代码。我的问题中的 OAuth 代码不起作用。这是我应该使用的代码：https://cloud.google.com/vertex-ai/docs/generative-ai/image/visual-question-answering#-python

注意，在我本地的 Conda Jupyter Notebooks 中进行实验时（https://github.com/CsabaConsulting/NextGenAI/blob/main/ImagenTest.ipynb）我仍然需要处理 ADC（应用程序默认凭据），请参阅 https://cloud.google.com/docs/authentication#auth-decision-tree 和 https://cloud.google.com/docs/authentication/application-default-credentials

在云功能中部署时，您需要建立正确的服务帐户。示例代码：https://github.com/CsabaConsulting/NextGenAI/tree/main/imagen_test

需求.txt：

functions-framework==3.*
google-cloud-aiplatform==1.35.*

main.py：

import base64
import functions_framework
import vertexai

from flask import jsonify
from vertexai.vision_models import ImageQnAModel, ImageTextModel, Image

PROJECT_ID = "gdg-demos"
LOCATION = "us-central1"

@functions_framework.http
def imagen_test(request):
    """HTTP Cloud Function.
    Args:
        request (flask.Request): The request object.
        <https://flask.palletsprojects.com/en/1.1.x/api/#incoming-request-data>
    Returns:
        The response text, or any set of values that can be turned into a
        Response object using `make_response`
        <https://flask.palletsprojects.com/en/1.1.x/api/#flask.make_response>.
    """
    request_json = request.get_json(silent=True)
    request_args = request.args

    if request_json and 'image' in request_json:
        image_b64 = request_json['image']
    elif request_args and 'image' in request_args:
        image_b64 = request_args['image']
    else:
        image_b64 = None

    if not image_b64:
        return jsonify(dict(data=[]))

    vertexai.init(project=PROJECT_ID, location=LOCATION)
    model = ImageQnAModel.from_pretrained("imagetext@001")

    image_binary = base64.b64decode(image_b64)
    image = Image(image_binary)
    answers = model.ask_question(
        image=image,
        question="Describe what is on the photo in great detail, be very verbose",
        number_of_results=3,
    )
    return jsonify(dict(data=answers))