长期运行Azure WebJob失败-客户端无法在指定的超时时间内完成操作

Question

我有一个运行时间较长的Azure WebJob（2-4h），大约90分钟后，由于存储异常，它始终失败。我正在使用WebJobs 2.3.0 SDK和WindowsAzure.Storage 9.3.3。

[09/17/2019 05:14:23 > b0c2e2: ERR ] 
[09/17/2019 05:14:23 > b0c2e2: ERR ] Unhandled Exception: Microsoft.WindowsAzure.Storage.StorageException: The client could not finish the operation within specified timeout. ---> System.TimeoutException: The client could not finish the operation within specified timeout.
[09/17/2019 05:14:23 > b0c2e2: ERR ]    --- End of inner exception stack trace ---
[09/17/2019 05:14:23 > b0c2e2: ERR ]    at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndExecuteAsync[T](IAsyncResult result) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Executor\Executor.cs:line 51
[09/17/2019 05:14:23 > b0c2e2: ERR ]    at Microsoft.WindowsAzure.Storage.Queue.CloudQueue.EndExists(IAsyncResult asyncResult) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Queue\CloudQueue.cs:line 994
[09/17/2019 05:14:23 > b0c2e2: ERR ]    at Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions.<>c__DisplayClass2`1.<CreateCallback>b__0(IAsyncResult ar) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Util\AsyncExtensions.cs:line 69
[09/17/2019 05:14:23 > b0c2e2: ERR ] --- End of stack trace from previous location where exception was thrown ---
[09/17/2019 05:14:23 > b0c2e2: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
[09/17/2019 05:14:23 > b0c2e2: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
[09/17/2019 05:14:23 > b0c2e2: ERR ]    at Microsoft.Azure.WebJobs.Host.Queues.Listeners.QueueListener.<ExecuteAsync>d__25.MoveNext()
[09/17/2019 05:14:23 > b0c2e2: ERR ] --- End of stack trace from previous location where exception was thrown ---
[09/17/2019 05:14:23 > b0c2e2: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
[09/17/2019 05:14:23 > b0c2e2: ERR ]    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
[09/17/2019 05:14:23 > b0c2e2: ERR ]    at Microsoft.Azure.WebJobs.Host.Timers.TaskSeriesTimer.<RunAsync>d__14.MoveNext()
[09/17/2019 05:14:23 > b0c2e2: ERR ] --- End of stack trace from previous location where exception was thrown ---
[09/17/2019 05:14:23 > b0c2e2: ERR ]    at Microsoft.Azure.WebJobs.Host.Timers.WebJobsExceptionHandler.<>c__DisplayClass3_0.<OnUnhandledExceptionAsync>b__0()
[09/17/2019 05:14:23 > b0c2e2: ERR ]    at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
[09/17/2019 05:14:23 > b0c2e2: ERR ]    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
[09/17/2019 05:14:23 > b0c2e2: ERR ]    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
[09/17/2019 05:14:23 > b0c2e2: ERR ]    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
[09/17/2019 05:14:23 > b0c2e2: ERR ]    at System.Threading.ThreadHelper.ThreadStart()
[09/17/2019 05:14:23 > b0c2e2: SYS ERR ] Job failed due to exit code -532462766
[09/17/2019 05:14:23 > b0c2e2: SYS INFO] Process went down, waiting for 0 seconds
[09/17/2019 05:14:23 > b0c2e2: SYS INFO] Status changed to PendingRestart

[这项工作对Azure存储没有做任何事情，并且从其他问题中我收集到这可能与WebJob将日志文件写入Azure存储有关，为此，我配置了一个自定义StorageClientFactory，服务器超时时间长，但这似乎没什么。

作业配置：

var config = new JobHostConfiguration()
config.Queues.MaxDequeueCount = 1; 
config.Queues.BatchSize = 1; 
ServicePointManager.DefaultConnectionLimit = int.MaxValue;
config.StorageClientFactory = new CustomStorageClientFactory();
var host = new JobHost(config);
host.RunAndBlock();



public class CustomStorageClientFactory : StorageClientFactory
{
    public override CloudBlobClient CreateCloudBlobClient(StorageClientFactoryContext context)
    {
        CloudBlobClient client = context.Account.CreateCloudBlobClient();
        client.DefaultRequestOptions.ServerTimeout = TimeSpan.FromHours(6);
        return client;
    }
}

Answer 1

这应该是Azure WebJob SDK 2.X的设计问题。在后端，它使用HttpWebRequest来访问存储API。问题是默认情况下每个服务器仅允许2个并发连接。因此，如果2个http连接被其他请求占用，则其他异步请求将超时。

解决方法是将DefaultConnectionLimit设置为更大的值，如下所示：

static void Main(string[] args)
{
    // Set this immediately so that it's used by all requests.
    ServicePointManager.DefaultConnectionLimit = Int32.MaxValue;

    var host = new JobHost();
    host.RunAndBlock();
}

您还可以简单地将Azure Web SDK升级到版本3.x，从而解决了此问题。

有关详情，请参阅Managing concurrent connections和https://github.com/Azure/azure-webjobs-sdk/issues/755#issuecomment-319094679。