InvalidDataException：超出了多部分主体长度限制16384

问题描述 投票：0回答：1

我想做什么：

我试图使用Postman上传一个带有文件和JSON blob的multipart/form-data到ASP.NET Core 2.2 APIController并将文件流式传输到磁盘上的临时文件而不是完全进入内存，因为文件的大小可能很大（20MB - 2GB）。我从大型文件示例开始跟随https://docs.microsoft.com/en-us/aspnet/core/mvc/models/file-uploads?view=aspnetcore-2.2的两个示例，但我也尝试使用相同的错误，类似但不同的堆栈跟踪测试小文件示例。服务器使用Kestrel。

大文件示例的堆栈跟踪（在调试器中捕获）：

Exception has occurred: CLR/System.IO.InvalidDataException
Exception thrown: 'System.IO.InvalidDataException' in System.Private.CoreLib.dll: 'Multipart body length limit 16384 exceeded.'
   at Microsoft.AspNetCore.WebUtilities.MultipartReaderStream.UpdatePosition(Int32 read)
   at Microsoft.AspNetCore.WebUtilities.MultipartReaderStream.<ReadAsync>d__36.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.AspNetCore.WebUtilities.StreamHelperExtensions.<DrainAsync>d__3.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.WebUtilities.MultipartReader.<ReadNextSectionAsync>d__20.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at LookupServiceAPI.Helpers.FileStreamingHelper.<StreamFile>d__1.MoveNext() in <hidden-path-to-project>\Helpers\FileStreamingHelper.cs:line 35

小文件示例的堆栈跟踪（作为响应返回，不会遇到任何断点或调试器异常捕获）：

System.IO.InvalidDataException: Multipart body length limit 16384 exceeded.
   at Microsoft.AspNetCore.WebUtilities.MultipartReaderStream.UpdatePosition(Int32 read)
   at Microsoft.AspNetCore.WebUtilities.MultipartReaderStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.WebUtilities.StreamHelperExtensions.DrainAsync(Stream stream, ArrayPool`1 bytePool, Nullable`1 limit, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.WebUtilities.MultipartReader.ReadNextSectionAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Http.Features.FormFeature.InnerReadFormAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Mvc.ModelBinding.FormValueProviderFactory.AddValueProviderAsync(ValueProviderFactoryContext context)
   at Microsoft.AspNetCore.Mvc.ModelBinding.CompositeValueProvider.CreateAsync(ActionContext actionContext, IList`1 factories)
   at Microsoft.AspNetCore.Mvc.ModelBinding.CompositeValueProvider.CreateAsync(ControllerContext controllerContext)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerBinderDelegateProvider.<>c__DisplayClass0_0.<<CreateBinderDelegate>g__Bind|0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.Invoke(HttpContext httpContext)
   at Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware.Invoke(HttpContext httpContext)
   at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)

这是我的基本控制器代码和大文件示例的帮助程序类：

FileStreamingHelper.cs

using System;
using System.Globalization;
using System.IO;
using System.Text;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Http.Features;
using Microsoft.AspNetCore.Http.Extensions;
using Microsoft.AspNetCore.Mvc.ModelBinding;
using Microsoft.AspNetCore.WebUtilities;
using Microsoft.Net.Http.Headers;

namespace LookupServiceAPI.Helpers
{
    public static class FileStreamingHelper
    {
        private static readonly FormOptions _defaultFormOptions = new FormOptions();

        public static async Task<FormValueProvider> StreamFile(this HttpRequest request, Stream targetStream)
        {
            if (!MultipartRequestHelper.IsMultipartContentType(request.ContentType))
            {
                throw new Exception($"Expected a multipart request, but got {request.ContentType}");
            }

            // Used to accumulate all the form url encoded key value pairs in the 
            // request.
            var formAccumulator = new KeyValueAccumulator();

            var boundary = request.GetMultipartBoundary();
            var reader = new MultipartReader(boundary, request.Body);
            reader.BodyLengthLimit = Int32.MaxValue;
            reader.HeadersLengthLimit = Int32.MaxValue;
            var section = await reader.ReadNextSectionAsync(); //EXCEPTION HERE
            while (section != null)
            {
                ContentDispositionHeaderValue contentDisposition;
                var hasContentDispositionHeader = ContentDispositionHeaderValue.TryParse(section.ContentDisposition, out contentDisposition);

                if (hasContentDispositionHeader)
                {
                    if (MultipartRequestHelper.HasFileContentDisposition(contentDisposition))
                    {
                        await section.Body.CopyToAsync(targetStream);
                    }
                    else if (MultipartRequestHelper.HasFormDataContentDisposition(contentDisposition))
                    {
                        // Content-Disposition: form-data; name="key"
                        //
                        // value

                        // Do not limit the key name length here because the 
                        // multipart headers length limit is already in effect.
                        var key = HeaderUtilities.RemoveQuotes(contentDisposition.Name);
                        var encoding = GetEncoding(section);
                        using (var streamReader = new StreamReader(
                            section.Body,
                            encoding,
                            detectEncodingFromByteOrderMarks: true,
                            bufferSize: 1024,
                            leaveOpen: true))
                        {
                            // The value length limit is enforced by MultipartBodyLengthLimit
                            var value = await streamReader.ReadToEndAsync();
                            if (String.Equals(value, "undefined", StringComparison.OrdinalIgnoreCase))
                            {
                                value = String.Empty;
                            }
                            formAccumulator.Append(key.Value, value); // For .NET Core <2.0 remove ".Value" from key

                            if (formAccumulator.ValueCount > _defaultFormOptions.ValueCountLimit)
                            {
                                throw new InvalidDataException($"Form key count limit {_defaultFormOptions.ValueCountLimit} exceeded.");
                            }
                        }
                    }
                }

                // Drains any remaining section body that has not been consumed and
                // reads the headers for the next section.
                section = await reader.ReadNextSectionAsync();
            }

            // Bind form data to a model
            var formValueProvider = new FormValueProvider(
                BindingSource.Form,
                new FormCollection(formAccumulator.GetResults()),
                CultureInfo.CurrentCulture);

            return formValueProvider;
        }

        private static Encoding GetEncoding(MultipartSection section)
        {
            MediaTypeHeaderValue mediaType;
            var hasMediaTypeHeader = MediaTypeHeaderValue.TryParse(section.ContentType, out mediaType);
            // UTF-7 is insecure and should not be honored. UTF-8 will succeed in 
            // most cases.
            if (!hasMediaTypeHeader || Encoding.UTF7.Equals(mediaType.Encoding) || mediaType.Encoding == null)
            {
                return Encoding.UTF8;
            }
            return mediaType.Encoding;
        }
    }
}

MultipartRequestHelper.cs

using System;
using System.IO;
using Microsoft.Net.Http.Headers;

namespace LookupServiceAPI.Helpers
{
    public static class MultipartRequestHelper
    {
        public static bool IsMultipartContentType(string contentType)
        {
            return !string.IsNullOrEmpty(contentType)
                   && contentType.IndexOf("multipart/", StringComparison.OrdinalIgnoreCase) >= 0;
        }

        public static bool HasFormDataContentDisposition(ContentDispositionHeaderValue contentDisposition)
        {
            // Content-Disposition: form-data; name="key";
            return contentDisposition != null
                   && contentDisposition.DispositionType.Equals("form-data")
                   && string.IsNullOrEmpty(contentDisposition.FileName.Value)
                   && string.IsNullOrEmpty(contentDisposition.FileNameStar.Value);
        }

        public static bool HasFileContentDisposition(ContentDispositionHeaderValue contentDisposition)
        {
            // Content-Disposition: form-data; name="myfile1"; filename="Misc 002.jpg"
            return contentDisposition != null
                   && contentDisposition.DispositionType.Equals("form-data")
                   && (!string.IsNullOrEmpty(contentDisposition.FileName.ToString())
                       || !string.IsNullOrEmpty(contentDisposition.FileNameStar.ToString()));
        }
    }
}

最小控制器：

[Route("api/v0.1/data/excel")]
[ApiController]
public class DataExcelController : ControllerBase
{
    [HttpPost, DisableRequestSizeLimit]
    public async Task<IActionResult> ImportExcel()
    {
        var processID = Guid.NewGuid();
        FormValueProvider multipartContent;
        string tempFilePath = Path.GetTempPath() + processID;
        using(var tempStream = System.IO.File.OpenWrite(tempFilePath))
        {
            multipartContent = await Request.StreamFile(tempStream);
        }
        /** Other unnecessary code **/
        return Ok();
    }
}

Startup.cs

namespace LookupServiceAPI
{
    public class Startup
    {
        public Startup(IConfiguration configuration)
        {
            Configuration = configuration;
        }

        public IConfiguration Configuration { get; }

        // This method gets called by the runtime. Use this method to add services to the container.
        public void ConfigureServices(IServiceCollection services)
        {
            services.AddMvc().SetCompatibilityVersion(CompatibilityVersion.Version_2_2);

            services.Configure<FormOptions>(x =>
            {
                x.MultipartHeadersLengthLimit = Int32.MaxValue;
                x.MultipartBoundaryLengthLimit = Int32.MaxValue;
                x.MultipartBodyLengthLimit = Int64.MaxValue;
                x.ValueLengthLimit = Int32.MaxValue;
                x.BufferBodyLengthLimit = Int64.MaxValue;
                x.MemoryBufferThreshold = Int32.MaxValue;
            });
        }

        // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
        public void Configure(IApplicationBuilder app, IHostingEnvironment env)
        {
            if (env.IsDevelopment())
            {
                app.UseDeveloperExceptionPage();
            }
            else
            {
                // The default HSTS value is 30 days. You may want to change this for production scenarios, see https://aka.ms/aspnetcore-hsts.
                app.UseHsts();
            }

            app.UseHttpsRedirection();
            app.UseMvc();
        }
    }
}

邮差配置图像（仅在图像中设置值，在标题中没有设置值）：Postman Configuration Image

邮差控制台输出：Postman Console Output

我试过的事情：

来自Multipart body length limit 16384 exceeded：

  • 设置MemoryBufferThreshold
  • 设置MultipartBodyLengthLimit
  • 确保邮递员配置标题没有Content-Type手动设置为multipart/form-data

来自Multipart body length limit exceeded exception：

  • 设置ValueLengthLimit
  • 二手[DisableRequestSizeLimit]

在哪里我认为问题是，但我不确定解决方法或导致问题的原因：https://github.com/aspnet/AspNetCore/blob/master/src/Http/WebUtilities/src/MultipartReader.cs#L48-L50

似乎我的请求的序言超出了为1024 * 16设置的DefaultHeadersLengthLimit（16384）的大小限制，但我不知道为什么会出现这种情况。或者，如果序言应该大于那个，如何解决它而不重新实现整个类或等待微软推出一个似乎没有出现在管道中的修复：https://github.com/aspnet/Mvc/issues/7019 https://github.com/aspnet/HttpAbstractions/issues/736

显然有人修复了他们的问题，这与我的非常相似（https://github.com/aspnet/Mvc/issues/5128#issuecomment-307675219）：https://github.com/aspnet/Mvc/issues/5128#issuecomment-307962922但我似乎无法弄清楚如何确定它是否适用。

希望这是足够的信息。如果没有，请告诉我您的需求，我很乐意提供或测试任何建议。我一直在研究这个并尝试我能找到的所有东西6个多小时。

c# asp.net-core postman asp.net-core-2.2
1个回答
0
投票

我已经解决了我的问题。原来这是我正在使用的URL。

为了解决我的问题，我意识到我正在发送到http端点而不是https端点，从而导致重定向。我将我的网址从http://localhost:5000/更改为https://localhost:5001/，一切都立即开始工作。

有趣的是，这在cURL中引起了一个问题，日志看起来像这样：

== Info: Connected to localhost (::1) port 5000 (#0)
=> Send header, 257 bytes (0x101)
0000: POST /api/v0.1/data/excel HTTP/1.1
0024: Host: localhost:5000
003a: User-Agent: curl/7.64.0
0053: Accept: */*
0060: cache-control: no-cache
0079: Content-Length: 13286446
0093: Content-Type: multipart/form-data; boundary=--------------------
00d3: ----7b12fc7773ed7878
00e9: Expect: 100-continue
00ff: 
== Info: Expire in 1000 ms for 0 (transfer 0xa6aa80)
<= Recv header, 33 bytes (0x21)
0000: HTTP/1.1 307 Temporary Redirect
<= Recv header, 37 bytes (0x25)
0000: Date: Tue, 09 Apr 2019 18:04:24 GMT
<= Recv header, 17 bytes (0x11)
0000: Server: Kestrel
<= Recv header, 19 bytes (0x13)
0000: Content-Length: 0
<= Recv header, 54 bytes (0x36)
0000: Location: https://localhost:5001/api/v0.1/data/excel
== Info: HTTP error before end of send, stop sending
<= Recv header, 2 bytes (0x2)
0000: 
== Info: Closing connection 0

更改端点也是固定的。

不知道为什么multipart/form-data上传因重定向而中断。如果有人有任何想法，我会很乐意学习。

最新问题
© www.soinside.com 2019 - 2024. All rights reserved.