NPGSQL开始文本导入在较大的CSV文件上引发异常

Question

编辑：经过更多测试后，此错误与HAPROXY直接相关。当我从堆栈中取出它时，它按预期方式工作。

我已经为这个问题苦苦挣扎了好几天。此代码确实适用于较小的CSV文件（25 KB），但是一旦尝试处理较大的文件（5000 KB），它将引发异常。

我的堆栈是下面的代码，从asp.net core 3.0，NPGSQL 4.1.1到HAPROXY，再到PostgreSQL 12实例。

这里是代码：

      string query = "COPY " + seedingDataApi.tableName + " FROM STDIN CSV ENCODING 'UTF8' QUOTE '\"'; ";
                using (var conn = postgresContext.GetDBConnectionPG())
                {

                    conn.Open();
                    using (var writer = conn.BeginTextImport(query))
                    {
                        _logger.LogInformation("Text import started");
                        using (StreamReader sr = new StreamReader(seedingDataApi.tableName + ".csv"))
                        {

                            _logger.LogInformation("Read stream started.");
                            while (!sr.EndOfStream)
                            {
                                string testString = "";

                                testString = sr.ReadLine();
                                while ("<EOL>" != testString.Substring(Math.Max(0, testString.Length - 5)))
                                {
                                    testString += sr.ReadLine();
                                }
                                testString = testString.Remove(testString.Length - 5);
                                writer.WriteLine(testString);
                            }

                        }
                    }
                }

应用程序引发的异常是：

 System.Exception: Npgsql.NpgsqlException (0x80004005): Exception while reading from stream
 ---> System.IO.EndOfStreamException: Attempted to read past the end of the stream.
   at Npgsql.NpgsqlReadBuffer.<>c__DisplayClass34_0.<<Ensure>g__EnsureLong|0>d.MoveNext()
   at Npgsql.NpgsqlReadBuffer.<>c__DisplayClass34_0.<<Ensure>g__EnsureLong|0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Npgsql.NpgsqlConnector.<>c__DisplayClass160_0.<<DoReadMessage>g__ReadMessageLong|0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Npgsql.NpgsqlConnector.<>c__DisplayClass160_0.<<DoReadMessage>g__ReadMessageLong|0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Npgsql.NpgsqlConnector.ReadMessage(DataRowLoadingMode dataRowLoadingMode)
   at Npgsql.NpgsqlRawCopyStream..ctor(NpgsqlConnector connector, String copyCommand)
   at Npgsql.NpgsqlConnection.BeginTextImport(String copyFromCommand)
   at MiniAppApiServer.Repositories.TableSeedRepo.TableSeedRepository.SeedTableDataAsync(TableSeedDataParams seedingDataApi, CancellationToken ct) in /source/MiniAppApiServer/Repositories/TableSeedRepo/TableSeedRepository.cs:line 306System.IO.EndOfStreamException: Attempted to read past the end of the stream.
   at Npgsql.NpgsqlReadBuffer.<>c__DisplayClass34_0.<<Ensure>g__EnsureLong|0>d.MoveNext()
   at MiniAppApiServer.Repositories.TableSeedRepo.TableSeedRepository.SeedTableDataAsync(TableSeedDataParams seedingDataApi, CancellationToken ct) in /source/MiniAppApiServer/Repositories/TableSeedRepo/TableSeedRepository.cs:line 339
   at MiniAppApiServer.Supervisors.TableSeedSupervisor.TableSeedSupervisor.SeedTableDataAsync(Stream data, String tablename, CancellationToken ct) in /source/MiniAppApiServer/Supervisors/TableSeedSupervisor/TableSeedSupervisor.cs:line 86
   at MiniAppApiServer.Controllers.TableSeedingController.PostTableDataSeed(IFormFile file, CancellationToken ct) in /source/MiniAppApiServer/Controllers/TableSeedingController.cs:line 114

非常感谢！

Answer 1

解决方案！问题出在HAProxy配置上。我更改了以下值：

timeout client          1m
timeout server          1m

至此：

timeout client          60m
timeout server          60m