Apache 每半夜都会自行重启，导致 apscheduler 任务无法完成

问题描述 投票：0回答：1

您好，我有一个使用 Apache 提供服务的 Pyramid wsgi Web 应用程序。 Web 应用程序有一个每小时作业，必须在第 0 分钟运行以获取时间敏感数据并写入我的 mysql 数据库。我注意到，有时（并非所有时候）数据可能不会在午夜 00:00:00 运行任务时写入数据库。查看日志，似乎 Apache 在每个午夜后不久都会重新启动，这可能会导致问题。

在 stackoverflow 搜索后，似乎 logrotate 可能是重启的罪魁祸首。但是，我还注意到 logrotate 由 crontab 调用，默认为上午 6:25，所以我不知道为什么重新启动发生在午夜。 （我的 Ubuntu 服务器没有安装 anacron）

这是 Apache 最近几天的日志文件

[Tue May 11 00:00:35.534821 2021] [mpm_event:notice] [pid 72273:tid 140034084613184] AH00489: Apache/2.4.41 (Ubuntu) mod_wsgi/4.6.8 Python/3.8 configured -- resuming normal operations
[Tue May 11 00:00:35.534867 2021] [core:notice] [pid 72273:tid 140034084613184] AH00094: Command line: '/usr/sbin/apache2'
.
.
.
[Wed May 12 00:00:00.029412 2021] [wsgi:error] [pid 72660:tid 140033624434432] 2021-05-12 00:00:00,029 INFO  [apscheduler.executors.default:123][ThreadPoolExecutor-0_0] Running job "XYZ (trigger: cron[minute='0'], next run at: 2021-05-12 01:00:00 HKT)" (scheduled at 2021-05-12 00:00:00+08:00)
[Wed May 12 00:00:00.621944 2021] [mpm_event:notice] [pid 72273:tid 140034084613184] AH00493: SIGUSR1 received.  Doing graceful restart
[Wed May 12 00:00:03.614647 2021] [wsgi:error] [pid 72660:tid 140033624434432] 2021-05-12 00:00:03,614 INFO  [apscheduler.executors.default:144][ThreadPoolExecutor-0_0] Job "XYZ (trigger: cron[minute='0'], next run at: 2021-05-12 01:00:00 HKT)" executed successfully

从上面的日志中有趣的是，我的 apscheduler 似乎仍然完成运行（数据库写入成功）并在 

Doing graceful restart
之后和创建新日志文件之前打印到日志（内容如下所示）

[Wed May 12 00:00:03.641095 2021] [mpm_event:notice] [pid 72273:tid 140034084613184] AH00489: Apache/2.4.41 (Ubuntu) mod_wsgi/4.6.8 Python/3.8 configured -- resuming normal operations
[Wed May 12 00:00:03.641146 2021] [core:notice] [pid 72273:tid 140034084613184] AH00094: Command line: '/usr/sbin/apache2'
.
.
.
[Thu May 13 00:00:00.032261 2021] [wsgi:error] [pid 95013:tid 140083656877824] 2021-05-13 00:00:00,032 INFO  [apscheduler.executors.default:123][ThreadPoolExecutor-0_0] Running job "XYZ (trigger: cron[minute='0'], next run at: 2021-05-13 01:00:00 HKT)" (scheduled at 2021-05-13 00:00:00+08:00)
[Thu May 13 00:00:03.764471 2021] [wsgi:error] [pid 95013:tid 140083656877824] 2021-05-13 00:00:03,764 INFO  [apscheduler.executors.default:144][ThreadPoolExecutor-0_0] Job "XYZ (trigger: cron[minute='0'], next run at: 2021-05-13 01:00:00 HKT)" executed successfully
[Thu May 13 00:00:34.829438 2021] [mpm_event:notice] [pid 95012:tid 140084121332800] AH00493: SIGUSR1 received.  Doing graceful restart

在上面的日志文件中，我的 apscheduler 作业在重新启动之前完成，因此我的数据库也被正确写入。

[Thu May 13 00:00:35.588354 2021] [mpm_event:notice] [pid 95012:tid 140084121332800] AH00489: Apache/2.4.41 (Ubuntu) mod_wsgi/4.6.8 Python/3.8 configured -- resuming normal operations
[Thu May 13 00:00:35.588433 2021] [core:notice] [pid 95012:tid 140084121332800] AH00094: Command line: '/usr/sbin/apache2'
.
.
.
[Fri May 14 00:00:00.020559 2021] [wsgi:error] [pid 2120:tid 140241617286912] 2021-05-14 00:00:00,020 INFO  [apscheduler.executors.default:123][ThreadPoolExecutor-0_0] Running job "XYZ (trigger: cron[minute='0'], next run at: 2021-05-14 01:00:00 HKT)" (scheduled at 2021-05-14 00:00:00+08:00)
[Fri May 14 00:00:00.558072 2021] [mpm_event:notice] [pid 2119:tid 140242151496768] AH00493: SIGUSR1 received.  Doing graceful restart

刚刚过去的午夜，作业还没有完成，数据库还没有写入。由于作业在完成之前突然终止，因此在午夜之前和之后也没有伴随的 

INFO  [apscheduler.executors.default:144][ThreadPoolExecutor-0_0] Job "XYZ (trigger: cron[minute='0'], next run at: xxxxxxx)" executed successfully
行写入任一日志中

[Fri May 14 00:00:03.588691 2021] [mpm_event:notice] [pid 2119:tid 140242151496768] AH00489: Apache/2.4.41 (Ubuntu) mod_wsgi/4.6.8 Python/3.8 configured -- resuming normal operations
[Fri May 14 00:00:03.588744 2021] [core:notice] [pid 2119:tid 140242151496768] AH00094: Command line: '/usr/sbin/apache2'
.
.
.
day hasn't ended yet

这是我的 crontab 文件，我认为它是标准的，并且规定每日作业应该在早上 6:25 运行，而不是午夜。

# /etc/crontab: system-wide crontab
# Unlike any other crontab you don't have to run the `crontab'
# command to install the new version when you edit this file
# and files in /etc/cron.d. These files also have username fields,
# that none of the other crontabs do.

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# Example of job definition:
# .---------------- minute (0 - 59)
# |  .------------- hour (0 - 23)
# |  |  .---------- day of month (1 - 31)
# |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...
# |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# |  |  |  |  |
# *  *  *  *  * user-name command to be executed
17 *    * * *   root    cd / && run-parts --report /etc/cron.hourly
25 6    * * *   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
47 6    * * 7   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
52 6    1 * *   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )

我的 Ubuntu 服务器没有安装 anacron 

ubuntu@xxx:~$ anacron --help

Command 'anacron' not found, but can be installed with:

sudo apt install anacron

logrotate 和 apache 有 cron.daily 任务

ubuntu@xxx:~$ ls -ln /etc/cron.daily/
total 40
-rwxr-xr-x 1 0 0  539 Apr 14  2020 apache2
-rwxr-xr-x 1 0 0  376 Dec  5  2019 apport
-rwxr-xr-x 1 0 0 1478 Apr  9  2020 apt-compat
-rwxr-xr-x 1 0 0  355 Dec 29  2017 bsdmainutils
-rwxr-xr-x 1 0 0 1187 Sep  6  2019 dpkg
-rwxr-xr-x 1 0 0  377 Jan 21  2019 logrotate
-rwxr-xr-x 1 0 0 1123 Feb 26  2020 man-db
-rwxr-xr-x 1 0 0 4574 Jul 18  2019 popularity-contest
-rwxr-xr-x 1 0 0  214 Dec  7 23:35 update-notifier-common

vi /etc/cron.daily/logrotate

#!/bin/sh

# skip in favour of systemd timer
if [ -d /run/systemd/system ]; then
    exit 0
fi

# this cronjob persists removals (but not purges)
if [ ! -x /usr/sbin/logrotate ]; then
    exit 0
fi

/usr/sbin/logrotate /etc/logrotate.conf
EXITVALUE=$?
if [ $EXITVALUE != 0 ]; then
    /usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"
fi
exit $EXITVALUE

vi /etc/cron.daily/apache2

#!/bin/sh

# run htcacheclean if set to 'cron' mode

set -e
set -u

type htcacheclean > /dev/null 2>&1 || exit 0
[ -e /etc/default/apache-htcacheclean ] || exit 0


# edit /etc/default/apache-htcacheclean to change this
HTCACHECLEAN_MODE=daemon
HTCACHECLEAN_RUN=auto
HTCACHECLEAN_SIZE=300M
HTCACHECLEAN_PATH=/var/cache/apache2/mod_cache_disk
HTCACHECLEAN_OPTIONS=""

. /etc/default/apache-htcacheclean

[ "$HTCACHECLEAN_MODE" = "cron" ] || exit 0

htcacheclean ${HTCACHECLEAN_OPTIONS}    \
                -p${HTCACHECLEAN_PATH}  \
                -l${HTCACHECLEAN_SIZE}

/etc/logrotate.conf 只是一个标准文件

# see "man logrotate" for details
# rotate log files weekly
weekly

# use the adm group by default, since this is the owning group
# of /var/log/syslog.
su root adm

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# use date as a suffix of the rotated file
#dateext

# uncomment this if you want your log files compressed
#compress

# packages drop log rotation information into this directory
include /etc/logrotate.d

# system-specific logs may be also be configured here.

apache2 包含在 logrotate.d 下

ubuntu@xxx:~$ ls -ln /etc/logrotate.d
total 52
-rw-r--r-- 1 0 0 120 Sep  6  2019 alternatives
-rw-r--r-- 1 0 0 442 Apr 14  2020 apache2
-rw-r--r-- 1 0 0 126 Dec  5  2019 apport
-rw-r--r-- 1 0 0 173 Apr  9  2020 apt
-rw-r--r-- 1 0 0  91 Nov  2  2020 bootlog
-rw-r--r-- 1 0 0 130 Jan 21  2019 btmp
-rw-r--r-- 1 0 0 112 Sep  6  2019 dpkg
-rw-r--r-- 1 0 0 845 Nov  7  2019 mysql-server
-rw-r--r-- 1 0 0 501 Mar  7  2019 rsyslog
-rw-r--r-- 1 0 0 119 Mar 31  2020 ubuntu-advantage-tools
-rw-r--r-- 1 0 0 178 Jan 22  2020 ufw
-rw-r--r-- 1 0 0 235 Jul 21  2020 unattended-upgrades
-rw-r--r-- 1 0 0 145 Feb 19  2018 wtmp

vi /etc/logrotate.d/apache2

/var/log/apache2/*.log {
        daily
        missingok
        rotate 14
        compress
        delaycompress
        notifempty
        create 640 root adm
        sharedscripts
        postrotate
                if invoke-rc.d apache2 status > /dev/null 2>&1; then \
                    invoke-rc.d apache2 reload > /dev/null 2>&1; \
                fi;
        endscript
        prerotate
                if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
                        run-parts /etc/logrotate.d/httpd-prerotate; \
                fi; \
        endscript
}

我只是想知道为什么我的 Apache 会在午夜重新启动，而根据 crontab 应该是早上 6:25，并更改时间以避免与我的第 0 分钟每小时工作发生冲突。谢谢！

apache apscheduler logrotate
1个回答
0
投票

通过以下方式检查您的 logrotate 启动模式是 logrotate.service 或 logrotate.timer： systemctl status logrotate.service

○ logrotate.service - Rotate log files
 Loaded: loaded (/lib/systemd/system/logrotate.service; static)
 Active: inactive (dead) since Mon 2024-01-29 00:00:04 UTC; 22h ago
 TriggeredBy: ● logrotate.timer
   Docs: man:logrotate(8)
         man:logrotate.conf(5)
 Process: 582357 ExecStart=/usr/sbin/logrotate /etc/logrotate.conf (code=exited, status=0/SUCCESS)

我相信它会由您系统上的 logrotate.timer 触发。 然后，您可以尝试 systemctl status logrotate.timer 来改善为什么在午夜轮换日志。

● logrotate.timer - Daily rotation of log files
 Loaded: loaded (/lib/systemd/system/logrotate.timer; enabled; vendor preset: enabled)
 Active: active (waiting) since Thu 2024-01-18 10:43:11 UTC; 1 week 4 days ago
 Trigger: Tue 2024-01-30 00:00:00 UTC; 1h 25min left
 Triggers: ● logrotate.service
   Docs: man:logrotate(8)
         man:logrotate.conf(5)

最后可以通过cat var/lib/logrotate/status

查看logrotate触发时的记录
最新问题
© www.soinside.com 2019 - 2024. All rights reserved.