Apache Storm 1.1.1 - 在哪里可以找到我的日志文件?

问题描述 投票:0回答:2

我检查过这些问题:

Logging from a storm bolt - where is it going?

而且解决方案不再适用了。

理论上,当我们启动storm.log.dir时,设置storm jar的系统变量。正如解决方案所示,您可以使用ps aux | grep storm.log.dir来搜索参数的值。

表明:

java -client -Ddaemon.name= -Dstorm.options= -Dstorm.home=/opt/apache-storm-1.1.1 -Dstorm.log.dir=/opt/apache-storm-1.1.1/logs .....

但是,当我去那里时,我有:

[root@xxx ~]# ls -la /opt/apache-storm-1.1.1/logs
total 58272
drwxr-xr-x  3 root root     4096 Jan 15 01:21 .
drwxr-xr-x 14 root root     4096 Oct 31 10:09 ..
-rw-r--r--  1 root root        0 Nov  3 10:46 access-logviewer.log
-rw-r--r--  1 root root        0 Oct 31 10:09 access-nimbus.log
-rw-r--r--  1 root root        0 Oct 31 10:09 access-supervisor.log
-rw-r--r--  1 root root        0 Nov 23 14:36 access-ui.log
-rw-r--r--  1 root root     8916 Nov 27 17:34 access-web-logviewer.log
-rw-r--r--  1 root root        0 Oct 31 10:09 access-web-nimbus.log
-rw-r--r--  1 root root        0 Oct 31 10:09 access-web-supervisor.log
-rw-r--r--  1 root root    31661 Nov 27 17:40 access-web-ui.log
-rw-r--r--  1 root root     2247 Nov  3 11:34 logviewer.log
-rw-r--r--  1 root root        0 Nov  3 10:46 logviewer.log.metrics
-rw-r--r--  1 root root    20690 Oct 31 10:09 nimbus.log
-rw-r--r--  1 root root        0 Oct 31 10:09 nimbus.log.metrics
-rw-r--r--  1 root root 46713727 Feb  6 17:13 supervisor.log
-rw-r--r--  1 root root  3143062 Jan 10 08:23 supervisor.log.1.gz
-rw-r--r--  1 root root  3104009 Jan 11 22:06 supervisor.log.2.gz
-rw-r--r--  1 root root  3103550 Jan 13 11:43 supervisor.log.3.gz
-rw-r--r--  1 root root  3103899 Jan 15 01:21 supervisor.log.4.gz
-rw-r--r--  1 root root        0 Oct 31 10:09 supervisor.log.metrics
-rw-r--r--  1 root root   401456 Nov 27 17:40 ui.log
-rw-r--r--  1 root root        0 Nov 23 14:36 ui.log.metrics
drwxr-xr-x 48 root root     4096 Feb 12 18:08 workers-artifacts

进入workers-artifacts,我有以节点命名的dirs,而在里面,我有worker.yaml

[root@mq1-acustats-process KafkaStormRadius-1-1518455061]# cd 1027

[root@mq1-acustats-process 1027]# ls -la
total 4
drwxr-xr-x 2 root root  24 Feb 12 18:04 .
drwxr-xr-x 8 root root  72 Feb 12 18:04 ..
-rw-r--r-- 1 root root 109 Feb 12 18:04 worker.yaml

[root@mq1-acustats-process 1027]# cat worker.yaml
worker-id: 33e6e77d-8a7b-48b2-b2d4-1be0af726c52
logs.users: []
logs.groups: []
topology.submitter.user: root

所以,这里没有日志。他们在哪?

我在worker.xml下有这个{STORM_DIR}/log4j2

<?xml version="1.0" encoding="UTF-8"?>
<!--
 Licensed to the Apache Software Foundation (ASF) under one or more
 contributor license agreements.  See the NOTICE file distributed with
 this work for additional information regarding copyright ownership.
 The ASF licenses this file to You under the Apache License, Version 2.0
 (the "License"); you may not use this file except in compliance with
 the License.  You may obtain a copy of the License at

     http://www.apache.org/licenses/LICENSE-2.0

 Unless required by applicable law or agreed to in writing, software
 distributed under the License is distributed on an "AS IS" BASIS,
 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
-->

<configuration monitorInterval="60" shutdownHook="disable">
<properties>
    <property name="pattern">%d{yyyy-MM-dd HH:mm:ss.SSS} %c{1.} %t [%p] %msg%n</property>
    <property name="patternNoTime">%msg%n</property>
    <property name="patternMetrics">%d %-8r %m%n</property>
</properties>
<appenders>
    <RollingFile name="A1"
        fileName="${sys:workers.artifacts}/${sys:storm.id}/${sys:worker.port}/${sys:logfile.name}"
        filePattern="${sys:workers.artifacts}/${sys:storm.id}/${sys:worker.port}/${sys:logfile.name}.%i.gz">
        <PatternLayout>
            <pattern>${pattern}</pattern>
        </PatternLayout>
        <Policies>
            <SizeBasedTriggeringPolicy size="100 MB"/> <!-- Or every 100 MB -->
        </Policies>
        <DefaultRolloverStrategy max="9"/>
    </RollingFile>
    <RollingFile name="radius"
        fileName="${sys:storm.log.dir}/radius.log"
        filePattern="${sys:storm.log.dir}/radius.log.%i.gz">
        <PatternLayout>
            <pattern>${pattern}</pattern>
        </PatternLayout>
        <Policies>
            <SizeBasedTriggeringPolicy size="50 MB"/> <!-- Or every 20 MB -->
        </Policies>
        <DefaultRolloverStrategy max="9"/>
    </RollingFile>
    <RollingFile name="STDOUT"
        fileName="${sys:storm.log.dir}/radius.out"
        filePattern="${sys:storm.log.dir}/radius.out.%i.gz">
        <PatternLayout>
            <pattern>${patternNoTime}</pattern>
        </PatternLayout>
        <Policies>
            <SizeBasedTriggeringPolicy size="50 MB"/> <!-- Or every 100 MB -->
        </Policies>
        <DefaultRolloverStrategy max="4"/>
    </RollingFile>
    <RollingFile name="STDERR"
        fileName="${sys:storm.log.dir}/radius.err"
        filePattern="${sys:storm.log.dir}/radius.err.%i.gz">
        <PatternLayout>
            <pattern>${patternNoTime}</pattern>
        </PatternLayout>
        <Policies>
            <SizeBasedTriggeringPolicy size="50 MB"/> <!-- Or every 100 MB -->
        </Policies>
        <DefaultRolloverStrategy max="4"/>
    </RollingFile>
    <RollingFile name="METRICS"
        fileName="${sys:storm.log.dir}/radius.metrics"
        filePattern="${sys:storm.log.dir}/radius.metrics.%i.gz">
        <PatternLayout>
            <pattern>${patternMetrics}</pattern>
        </PatternLayout>
        <Policies>
            <SizeBasedTriggeringPolicy size="2 MB"/>
        </Policies>
        <DefaultRolloverStrategy max="9"/>
    </RollingFile>
    <Syslog name="syslog" format="RFC5424" charset="UTF-8" host="localhost" port="514"
        protocol="UDP" appName="[${sys:storm.id}:${sys:worker.port}]" mdcId="mdc" includeMDC="true"
        facility="LOCAL5" enterpriseNumber="18060" newLine="true" exceptionPattern="%rEx{full}"
        messageId="[${sys:user.name}:${sys:logging.sensitivity}]" id="storm" immediateFail="true" immediateFlush="true"/>
</appenders>
<loggers>
    <root level="info"> <!-- We log everything -->
        <appender-ref ref="A1"/>
        <appender-ref ref="syslog"/>
    </root>
    <Logger name="org.apache.storm.metric.LoggingMetricsConsumer" level="info" additivity="false">
        <appender-ref ref="METRICS"/>
    </Logger>
    <Logger name="com.joestelmach.natty" level="error" additivity="false">
        <appender-ref ref="radius"/>
    </Logger>
    <Logger name="STDERR" level="INFO">
        <appender-ref ref="STDERR"/>
        <appender-ref ref="syslog"/>
    </Logger>
    <Logger name="STDOUT" level="INFO">
        <appender-ref ref="STDOUT"/>
        <appender-ref ref="syslog"/>
    </Logger>
</loggers>
</configuration>

但是,我没有看到在radius.log下创建的${sys:storm.log.dir}

为什么?

顺便说说:

  • cluster.xmlworker.xml之间的区别?
log4j2 apache-storm
2个回答
1
投票

我的猜测是没有为worker JVM正确设置storm.log.dir变量。请记住,运行storm jar时启动的JVM与运行拓扑的JVM不同。

为了为您的worker设置VM选项,您可以使用storm.yaml中的worker.childopts变量来全局设置选项(如果这样做,您可能希望确保从https://github.com/apache/storm/blob/v1.1.1/conf/defaults.yaml#L171复制默认值)或topology.worker拓扑配置中的.childopts为每个拓扑设置它们。

例如,我使用以下配置将日志打印到E:\ testLogs:

storm.yaml

worker.childopts: "-Dstorm.log.dir=E:\\testLogs"

worker.xml

...
<RollingFile name="A1"
    fileName="${sys:storm.log.dir}/worker.log"
    filePattern="${sys:workers.artifacts}/${sys:storm.id}/${sys:worker.port}/${sys:logfile.name}.%i.gz">
    <PatternLayout>
        <pattern>${pattern}</pattern>
    </PatternLayout>
    <Policies>
        <SizeBasedTriggeringPolicy size="100 MB"/> <!-- Or every 100 MB -->
    </Policies>
    <DefaultRolloverStrategy max="9"/>
</RollingFile>
...

其中,worker.xml的其余部分是与Storm 1.1.1一起发布的默认值。

关于cluster.xml和worker.xml,cluster.xml是Storm守护程序(Nimbus,Supervisor,UI等)的日志配置,而worker.xml是工作进程(运行拓扑组件的进程)的日志配置


0
投票

我已经阅读了另一个团队留下的文档并找到了这些行....我有一个未启动的风暴服务。

systemctl start storm-supervisor

检查/etc/systemd/system/下的服务定义(我使用的是CentOS,所以我使用的是systemd),它显示为:

[Unit]
Description=Nimbus Service
After=network.target

[Service]
Type=simple
Restart=always
RestartSec=0s
ExecStart=/opt/apache-storm/bin/storm supervisor

[Install]
WantedBy=multi-user.target

启动后,我现在看到在正确的目录下创建的日志文件。

值得一提的是,我还改变了另外两件事:

  • 使用slf4j而不是log4j
  • 更改log4j2.properties以使用绝对路径(在/tmp下)而不是像{sys:storm.log.dir}那样读取系统变量以便丢弃可变因子。
© www.soinside.com 2019 - 2024. All rights reserved.