使用Boost的Zstd过滤流从压缩日志文件（log4j）中读取新添加的内容

Question

我目前正在开发一个 C++ 程序，旨在跟踪存储在磁盘上的实时日志文件。问题是日志文件是使用 Zstd 压缩的。我一直在使用 Boost (1.83) Zstd 过滤流进行解压缩，但遇到了障碍。过滤流似乎没有拾取新添加到日志文件中的内容。

我已经尝试解决此问题，但运气不佳。有谁有这方面的经验或知道我可以咨询指导的任何资源吗？我已经添加了我的代码如下所示。

预先感谢您提供的任何帮助或建议！

#include <iostream>
#include <string>
#include <boost/iostreams/filtering_streambuf.hpp>
#include <boost/iostreams/filter/zlib.hpp> // Example filter (can be replaced with any filter)
#include <fstream>
#include <zstd.h>

namespace io = boost::iostreams;

int main() {
    // Create a filtering_istreambuf with a zlib decompressor
    io::filtering_streambuf<io::input> in;
    in.push(io::zstd_decompressor());
    std::ifstream file("compressed_file.txt.zst", std::ios_base::in | std::ios_base::binary);
    in.push(file);
    // Construct an istream from the underlying streambuf
    std::istream inputStream(&in);
    // Read lines from the input stream using std::getline()

    while (true) {

        std::string line;
        while (std::getline(inputStream, line)) {
            std::cout << "Line: " << line << std::endl;
        }

        if (!file.eof()) {
            std::cout << "EOF reached" << std::flush;
            break; // Ensure end of read was EOF.
        }
        file.clear();
    }
    
    return 0;
}

Answer 1

我认为减压不是你的问题。这只是“体现”你的问题。即使不解压，您也会遇到同样的问题：流在 EOF 处停止。 它看起来并不像

iostreams

有像尾巴这样的设施，但你总是可以使用......

tail

：

住在Coliru

#include <boost/iostreams/filter/gzip.hpp>
#include <boost/iostreams/filtering_stream.hpp>
#include <boost/process.hpp>
#include <iostream>
using namespace std::string_literals;
namespace bp = boost::process;
namespace io = boost::iostreams;

int main() {
    bp::ipstream monitor;
    bp::child    tail("/usr/bin/tail",                        //
                      std::vector{"compressed_file.txt.gz"s}, //
                      bp::std_in.null(),                      //
                      bp::std_out = monitor);

    io::filtering_istream filter;
    filter.push(io::gzip_decompressor{});
    filter.push(monitor);

    for (std::string line; getline(filter, line);)
        std::cout << "line: " << line << "\n";

    tail.wait();
}

测试

gzip -k main.cpp; mv main.cpp.gz compressed_file.txt.gz
g++ -std=c++17 -O2 -Wall -pedantic -pthread main.cpp -lz -lboost_iostreams 
./a.out

打印预期：

line: #include <boost/iostreams/filter/gzip.hpp>
line: #include <boost/iostreams/filtering_stream.hpp>
line: #include <boost/process.hpp>
line: #include <iostream>
line: using namespace std::string_literals;
line: namespace bp = boost::process;
line: namespace io = boost::iostreams;
line: 
line: int main() {
line:     bp::ipstream monitor;
line:     bp::child    tail("/usr/bin/tail",                        //
line:                       std::vector{"compressed_file.txt.gz"s}, //
line:                       bp::std_in.null(),                      //
line:                       bp::std_out = monitor);
line: 
line:     io::filtering_istream filter;
line:     filter.push(io::gzip_decompressor{});
line:     filter.push(monitor);
line: 
line:     for (std::string line; getline(filter, line);)
line:         std::cout << "line: " << line << "\n";
line: 
line:     tail.wait();
line: }

奖金

当然，删除对 boost Boost IOstream 和 zlib 的链接依赖请考虑：

#include <boost/process.hpp>
#include <iostream>
namespace bp = boost::process;

int main() {
    bp::pstream tail_out, unzip_out;
    bp::child                                       //
        tail("/usr/bin/tail",                       //
             bp::std_in < "compressed_file.txt.gz", //
             bp::std_out = tail_out),               //
        unzip("/usr/bin/zcat",                      //
              bp::std_in  = tail_out,               //
              bp::std_out = unzip_out);

    for (std::string line; getline(unzip_out, line);)
        std::cout << "line: " << line << "\n";

    return unzip.exit_code();
}

还是一样的效果。当然更简单：

#include <boost/process.hpp>
#include <iostream>
namespace bp = boost::process;

int main() {
    bp::pstream stream;
    bp::child pipeline("tail | zcat", bp::shell, bp::std_in < "compressed_file.txt.gz", bp::std_out = stream);

    for (std::string line; getline(stream, line);)
        std::cout << "line: " << line << "\n";

    return pipeline.exit_code();
}

印刷

line: #include <boost/process.hpp>
line: #include <iostream>
line: namespace bp = boost::process;
line: 
line: int main() {
line:     bp::pstream stream;
line:     bp::child pipeline("tail | zcat", bp::shell, bp::std_in < "compressed_file.txt.gz", bp::std_out = stream);
line: 
line:     for (std::string line; getline(stream, line);)
line:         std::cout << "line: " << line << "\n";
line: 
line:     return pipeline.exit_code();
line: }