水槽Appender的问题:当水槽代理重记录期间下降了应用程序线程被卡住

问题描述 投票:0回答:1

目前,我们正在使用log4j的下方坛子

log4j的核心-2.6.2,log4j的-水槽-OF-2.6.2

<Flume name="aggregatorApp" compress="false" type="Avro"
ignoreExceptions="true" batchSize="10" blocking="false" >
<Agent host="${flumeHostPrimary}" port="${flumePortPrimary}" />
<Agent host="${flumeHostSecondary}" port="${flumePortSecondary}" />
<FlumeEventFactory logType="APPLICATION"/>
<PatternLayout header="app" pattern="[%-5p] %d %c %X{correlationId} - %m" />

<AsyncLogger name="com.xyz.abc" level="debug"
additivity="false" blocking="false">
<AppenderRef ref="IDRESTSERVICELOG" />
<AppenderRef ref="aggregatorApp" level="INFO" />
<AppenderRef ref="CONSOLE" level="INFO"/>
</AsyncLogger>
  1. 当水槽代理启动并运行应用程序运行流畅,并记录的事件被成功地发布到水槽代理商
  2. 如果水槽剂为下降一段时间后,应用程序线程都被卡住以下例外情况

Exception in thread "elasticsearch[_client_][generic][T#3]" java.lang.OutOfMemoryError: GC overhead limit exceeded
2019-02-01 07:43:23,618 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] WARN  org.apache.catalina.valves.StuckThreadDetectionValve- Thread "http-apr-8080-exec-7" (id=167) has been active for 20,392 milliseconds (since 2/1/19 7:42 AM) to serve the same request for htpps://xyz//directories/v1.0/search and may be stuck (configured threshold for this StuckThreadDetectionValve is 20 seconds). There is/are 1 thread(s) in total that are monitored by this Valve and may be stuck.
2019-02-01 07:46:18,619 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] ERROR org.apache.catalina.core.ContainerBase- Unexpected death of background thread ContainerBackgroundProcessor[StandardEngine[Catalina]]
java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "http-apr-8080-exec-42" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "http-apr-8080-exec-50" Exception in thread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" Exception in thread "I/O dispatcher 23" 
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "ContainerBackgroundProcessor[StandardEngine[Catalina]]"
Exception in thread "http-apr-8080-exec-38" Exception in thread "http-apr-8080-exec-46" Exception in thread "http-apr-8080-exec-53" Exception in thread "I/O dispatcher 17" 
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "http-apr-8080-exec-31"

上述异常重负载(每个请求1MB日志)注过程中发生的事情:我们正在做负载测试

随着应用程序在很大程度上记录,但水槽附加器不能用同样的速度消耗。我们怀疑Log4j的环形缓冲区快满这是阻止应用程序线程和创建内存不足异常。

我们尝试了几个考虑有什么用Log4J的追加程序的行为,

-Dlog4j2.AsyncQueueFullPolicy =丢弃-Dlog4j2.enable.threadlocals =真-Dlog4j2.enable.direct.encoders =真

但上述系统性能是没有用的。我们都失去了一些东西?

java performance log4j2 flume flume-ng
1个回答
0
投票

我会同意,RingBuffer为AsyncLogger导致你耗尽内存。环形缓冲区的默认大小为256K条目。什么是JVM的堆大小?

另一种方法将是使用任一FlumeAppender的嵌入式或持久性的变化。这些将转发到水槽代理之前缓存在本地文件通道的事件。这将允许你控制多久,服务器可以通过调整多少磁盘空间可容忍停运。你依然会得到异步记录器的优点,但是从断电保护。

© www.soinside.com 2019 - 2024. All rights reserved.