AWS EC2磁盘已满，无法释放空间

Question

我正在使用EC2实例来运行节点应用程序。一段时间后，我登录到服务器，只是发现服务器磁盘空间不足。调试后，我意识到日志正在占用空间。我删除了3.3Gb日志文件。但是，即使清除后也没有空间。我该怎么办？

这是我运行的命令：

ubuntu@app1:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            488M     0  488M   0% /dev
tmpfs           100M   11M   89M  11% /run
/dev/xvda1      7.7G  7.7G     0 100% /
tmpfs           496M  8.0K  496M   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           496M     0  496M   0% /sys/fs/cgroup
tmpfs           100M     0  100M   0% /run/user/1001
tmpfs           100M     0  100M   0% /run/user/1000

---

ubuntu@app1:~$ sudo du -h --max-depth=1 / | sort -n
0   /proc
0   /sys
4.0K    /lib64
4.0K    /media
4.0K    /mnt
4.0K    /srv
8.0K    /dev
8.0K    /snap
16K /lost+found
24K /root
800K    /tmp
6.4M    /etc
11M /run
14M /sbin
16M /bin
246M    /boot
331M    /home
397M    /opt
429M    /var
538M    /lib
2.1G    /usr
3.7G    /data
7.7G    /

我删除了/data中的3.3G日志文件，然后再次运行了du

ubuntu@app1:~$ sudo du -h --max-depth=1 / | sort -h
0   /proc
0   /sys
4.0K    /lib64
4.0K    /media
4.0K    /mnt
4.0K    /srv
8.0K    /dev
8.0K    /snap
16K /lost+found
24K /root
800K    /tmp
6.4M    /etc
11M /run
14M /sbin
16M /bin
246M    /boot
331M    /home
352M    /data
397M    /opt
429M    /var
538M    /lib
2.1G    /usr
4.4G    /

---

ubuntu@app1:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            488M     0  488M   0% /dev
tmpfs           100M   11M   89M  11% /run
/dev/xvda1      7.7G  7.7G     0 100% /
tmpfs           496M  8.0K  496M   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           496M     0  496M   0% /sys/fs/cgroup
tmpfs           100M     0  100M   0% /run/user/1001
tmpfs           100M     0  100M   0% /run/user/1000

尽管/data目录现在减少到352M，但df仍显示100％的磁盘利用率。我想念什么？

参考此答案https://unix.stackexchange.com/a/253655/47050，这是strace的输出

ubuntu@app1:~$ strace -e statfs df /
statfs("/", {f_type="EXT2_SUPER_MAGIC", f_bsize=4096, f_blocks=2016361, f_bfree=4096, f_bavail=0, f_files=1024000, f_ffree=617995, f_fsid={2136106470, -680157247}, f_namelen=255, f_frsize=4096, f_flags=4128}) = 0
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/xvda1       8065444 8049060         0 100% /
+++ exited with 0 +++

UPDATE

我跑了

sudo lsof | grep deleted

并找到

node\x20/ 22318           deploy   12w      REG              202,1 3541729280     791684 /data/app/shared/logs/production.log (deleted)
node\x20/ 22318           deploy   13w      REG              202,1 3541729280     791684 /data/app/shared/logs/production.log (deleted)
node\x20/ 22318           deploy   14w      REG              202,1 3541729280     791684 /data/app/shared/logs/production.log (deleted)
node\x20/ 22318           deploy   15w      REG              202,1 3541729280     791684 /data/app/shared/logs/production.log (deleted)
node\x20/ 22318           deploy   16w      REG              202,1 3541729280     791684 /data/app/shared/logs/production.log (deleted)

如何释放这些文件？

更新2

ubuntu@app1:~$ sudo ls -l /proc/22318/fd
total 0
lrwx------ 1 deploy deploy 64 Apr  6 10:05 0 -> socket:[74749956]
lrwx------ 1 deploy deploy 64 Apr  6 10:05 1 -> socket:[74749958]
lr-x------ 1 deploy deploy 64 Apr  6 10:05 10 -> /dev/null
l-wx------ 1 deploy deploy 64 Apr  6 10:05 12 -> /data/app/shared/logs/production.log (deleted)
l-wx------ 1 deploy deploy 64 Apr  6 10:05 13 -> /data/app/shared/logs/production.log (deleted)
l-wx------ 1 deploy deploy 64 Apr  6 10:05 14 -> /data/app/shared/logs/production.log (deleted)
l-wx------ 1 deploy deploy 64 Apr  6 10:05 15 -> /data/app/shared/logs/production.log (deleted)
l-wx------ 1 deploy deploy 64 Apr  6 10:05 16 -> /data/app/shared/logs/production.log (deleted)
l-wx------ 1 deploy deploy 64 Apr  6 10:05 17 -> /data/app/shared/logs/production.log (deleted)
l-wx------ 1 deploy deploy 64 Apr  6 10:05 18 -> /data/app/shared/logs/production.log (deleted)
l-wx------ 1 deploy deploy 64 Apr  6 10:05 19 -> /data/app/shared/logs/production.log (deleted)
lrwx------ 1 deploy deploy 64 Apr  6 10:05 2 -> socket:[74749960]
l-wx------ 1 deploy deploy 64 Apr  6 10:05 20 -> /data/app/shared/logs/production.log (deleted)
lrwx------ 1 deploy deploy 64 Apr  6 10:05 21 -> socket:[74750302]
lrwx------ 1 deploy deploy 64 Apr  6 10:05 22 -> socket:[74750303]
lrwx------ 1 deploy deploy 64 Apr  6 10:05 3 -> socket:[74749962]
lrwx------ 1 deploy deploy 64 Apr  6 10:05 4 -> anon_inode:[eventpoll]
lr-x------ 1 deploy deploy 64 Apr  6 10:05 5 -> pipe:[74749978]
l-wx------ 1 deploy deploy 64 Apr  6 10:05 6 -> pipe:[74749978]
lr-x------ 1 deploy deploy 64 Apr  6 10:05 7 -> pipe:[74749979]
l-wx------ 1 deploy deploy 64 Apr  6 10:05 8 -> pipe:[74749979]
lrwx------ 1 deploy deploy 64 Apr  6 10:05 9 -> anon_inode:[eventfd]

---

ubuntu@app1:~$ ps aux | grep node
deploy   22318  0.0 12.7 1277192 129844 ?      Ssl   2019 173:38 node /data/app/releases/20180904094535/app.js
ubuntu   30665  0.0  0.0  12944   972 pts/0    S+   10:09   0:00 grep --color=auto node

Answer 1

文件由节点应用程序保留。确定使用：

sudo lsof | grep deleted

重新启动节点应用程序解决了我的问题。

查找节点进程ID ps aux | grep node。然后使用kill -9 <process_id>杀死节点服务器。最后重启节点服务器。在我的情况下，pm2自动重新启动节点。