在内存不饱和的情况下将巨大的nvarchar(max)读取为UTF-16流

Question

我正在开发一个程序，该程序从 sql server nvarchar(max) 字段读取文本，并需要将其写入 zip 内的 UTF-16 文本文件。

我无法加载该字段中包含的字符串，因为它的重量 >1GB 并且会使内存饱和

我想做的是将流直接放入即时创建的 .zip 中

我找到了一个方法，但我没有得到有效的utf-16文件，因为我直接读取sql字段的字节。你还有其他方法吗？

using (SqlDataReader Rd = DbMan.GetDbReader(command, CommandBehavior.SequentialAccess) as SqlDataReader) //Opens a datareader in SequentialAccess fro not impacting on memory
{
    using (MemoryStream MSZip = new MemoryStream()) //This stream contains the .zip created on the fly
    {
        using (var Zip = new ZipArchive(MSZip, ZipArchiveMode.Create)) //Creating .zip
        {
            while (Rd.Read())
            {
                var FileJsonInsideZip = Zip.CreateEntry("MyFile${Rd.GetString(0)}.json", CompressionLevel.Fastest); //Creating the empty file inside .zip

            using (var StreamFileInsideZip = FileJsonInsideZip.Open()) //This stream is the file created inside the .zip
            {
                byte[] buffer = new byte[8096]; //Reading 8096 bytes and adds progressively to zip, to not saturate the memory
                long offset = 0;
                long read;
                while ((read = Rd.GetBytes(1, offset, buffer, 0, buffer.Length)) > 0)
                {
                    offset += read;
                    StreamFileInsideZip.Write(buffer, 0, (int)read);
                }
                
            }

        }

    }
}
}

我也尝试使用 TextReader，这种方法会生成有效的 UTF-16 文件，但会使内存饱和

byte[] sUtf16Bytes = Encoding.Unicode.GetBytes(Rd.GetTextReader(1).ReadToEnd());
StreamFileInsideZip.Write(sUtf16Bytes, 0, sUtf16Bytes.Length); 

Answer 1

您应该将缓冲区移到循环之外。

也许还可以将

MemoryStream

初始化为较大的缓冲区大小以避免复制。

byte[] buffer = new byte[8096]; //Reading 8096 bytes and adds progressively to zip, to not saturate the memory

using SqlDataReader Rd = (SqlDataReader)DbMan.GetDbReader(command, CommandBehavior.SequentialAccess);
using MemoryStream MSZip = new MemoryStream() //This stream contains the .zip created on the fly
using (var Zip = new ZipArchive(MSZip, ZipArchiveMode.Create, leaveOpen: true)) //Creating .zip, leave it open so you can use MSZip after
{
    while (Rd.Read())
    {
        var FileJsonInsideZip = Zip.CreateEntry($"MyFile{Rd.GetString(0)}.json", CompressionLevel.Fastest); //Creating the empty file inside .zip

        using var StreamFileInsideZip = FileJsonInsideZip.Open(); //This stream is the file created inside the .zip
        long offset = 0;
        long read;
        while ((read = Rd.GetBytes(1, offset, buffer, 0, buffer.Length)) > 0)
        {
            offset += read;
            StreamFileInsideZip.Write(buffer, 0, (int)read);
        }
    }
}
MSZip.Position = 0;  // reset position of memorystream to copy somewhere else