AWS lambda sdk 错误：“TooManyRequestsException”

Question

我正在构建一项服务，通过电子邮件向订阅者发送抓取的数据。

服务架构很简单：

1. “Lambda A”获取目标 URL。 （网址总数只有14个）
2. “Lambda A”触发“Lambda B”，实际上使用 AWS-SDK 抓取数据。
3. “Lambda A”等待每个“Lambda B”的执行，以将整个抓取的数据保存在 AWS S3 中。
4. 当对象保存在特定的S3中时，“Lambda C”被触发，并开始向订阅者发送电子邮件。

但是，问题出现在第2步。这是我的“Lambda A”代码。

const databaseUrl = process.env.DATABASE_URL;
const mysql = require('mysql2/promise');
const { LambdaClient, InvokeCommand } = require('@aws-sdk/client-lambda');

const client = new LambdaClient();

// Connect to DB and get URLs.
const connect = async (url) => {
  try {
    const connection = await mysql.createConnection(url);
    const [rows, fields] = await connection.query("select * from source");
    const result = rows.map((source) => {
      return {
        name: source.name,
        url: source.url,
        description: source.description,
      };
    });
    connection.end();
    return result;
  } catch (err) {
    return "err";
  }
};

/**
 * This function invokesa lambda function synchronously to perform subsequent logic(Trigger Send Email)
 */
const invoke = async (client, source) => {
  const params = {
    FunctionName: 'scraper',
    InvocationType: 'RequestResponse',
    Payload: JSON.stringify(source)
  };
  return await client.send(new InvokeCommand(params));
};

const handler = async (event) => {
  const sources = await connect(databaseUrl);
  
  const lambdas = sources.map((source) => invoke(client, source));
  
  const results = await Promise.all(lambdas);
  
  const decoder = new TextDecoder('utf-8');
  
  for (const result of results) {
    const payload = result.Payload;
    console.log(decoder.decode(payload));
  }
  
  return JSON.stringify(results);
};

module.exports = {
  handler
}

这是来自 aws lambda 的错误跟踪。

{
    "errorType": "TooManyRequestsException",
    "errorMessage": "Rate Exceeded.",
    "name": "TooManyRequestsException",
    "$fault": "client",
    "$metadata": {
        "httpStatusCode": 429,
        "requestId": "d9a2b812-2c76-4cda-b688-201824c4194b",
        "attempts": 3,
        "totalRetryDelay": 762
    },
    "Type": "User",
    "Reason": "ConcurrentInvocationLimitExceeded",
    "stack": [
        "TooManyRequestsException: Rate Exceeded.",
        "    at de_TooManyRequestsExceptionRes (/var/runtime/node_modules/@aws-sdk/client-lambda/dist-cjs/protocols/Aws_restJson1.js:5568:23)",
        "    at de_InvokeCommandError (/var/runtime/node_modules/@aws-sdk/client-lambda/dist-cjs/protocols/Aws_restJson1.js:3344:25)",
        "    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)",
        "    at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24",
        "    at async /var/runtime/node_modules/@aws-sdk/middleware-signing/dist-cjs/awsAuthMiddleware.js:30:20",
        "    at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/middleware-retry/dist-cjs/retryMiddleware.js:27:46",
        "    at async /var/runtime/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:7:26",
        "    at async invoke (/var/task/index.js:31:10)",
        "    at async Promise.all (index 8)",
        "    at async Runtime.handler (/var/task/index.js:51:19)"
    ]
}

我从此文档中发现了有关“ConcurrentInitationLimitExceeded”错误 - Invoke - AWS Lambda。以及本文档中有关 lambda 缩放的其他信息 - Lambda 函数缩放。

文档说，出现这样的错误的原因是因为不能同时执行超过1000个左右的lambda。然而我的“Lambda A”只调用了 14 个额外的 lambda 函数。我的问题（架构或代码）有解决方案吗？

Answer 1

根据配额页面：

新的 AWS 账户减少了并发和内存配额。 AWS 会根据您的使用情况自动提高这些配额。

在 AWS Lambda 控制台仪表板中检查您的实际配额。初始配额实际为 10 个的情况并不少见，并且会随着时间的推移而增加。