Prometheus表单prometheus-community图表启动失败（特别是prometheus-server）

Question

大约一个月后，我向我的普罗米修斯添加了更多警报，并希望升级现有的 helm-chart。然而，升级后，除了 prometheus-server 容器之外，一切看起来都很正常。这是日志：

kubectl logs prometheus-server-58c58c58cc-xhlct -c prometheus-server -n monitoring
ts=2022-06-13T14:50:54.109Z caller=main.go:516 level=info msg="Starting Prometheus" version="(version=2.34.0, branch=HEAD, revision=881111fec4332c33094a6fb2680c71fffc427275)"
ts=2022-06-13T14:50:54.109Z caller=main.go:521 level=info build_context="(go=go1.17.8, user=root@121ad7ea5487, date=20220315-15:18:00)"
ts=2022-06-13T14:50:54.109Z caller=main.go:522 level=info host_details="(Linux 5.4.0-100-generic #113-Ubuntu SMP Thu Feb 3 18:43:29 UTC 2022 x86_64 prometheus-server-58c58c58cc-xhlct (none))"
ts=2022-06-13T14:50:54.109Z caller=main.go:523 level=info fd_limits="(soft=1048576, hard=1048576)"
ts=2022-06-13T14:50:54.109Z caller=main.go:524 level=info vm_limits="(soft=unlimited, hard=unlimited)"
ts=2022-06-13T14:50:54.111Z caller=web.go:540 level=info component=web msg="Start listening for connections" address=0.0.0.0:9090
ts=2022-06-13T14:50:54.112Z caller=main.go:937 level=info msg="Starting TSDB ..."
ts=2022-06-13T14:50:54.112Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1653760808938 maxt=1653825600000 ulid=01G48C5KWQ1K30MFC4C3JQY4BP
ts=2022-06-13T14:50:54.112Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1653825608938 maxt=1653890400000 ulid=01G4A9Z7JQ2Y6XFQQBCYVRFH09
ts=2022-06-13T14:50:54.112Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1653890405501 maxt=1653955200000 ulid=01G4C7RQCWZP4BHKBE8JNDJKME
ts=2022-06-13T14:50:54.112Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1653955205501 maxt=1654020000000 ulid=01G4E5J8G761VCEGYZF1TD4QMN
ts=2022-06-13T14:50:54.112Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1654020005501 maxt=1654084800000 ulid=01G4G3BTMZT7WEQB8N364JCWYF
ts=2022-06-13T14:50:54.113Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1654084805501 maxt=1654149600000 ulid=01G4J15B267JNEY4TGWR42KSVG
ts=2022-06-13T14:50:54.113Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1654149608938 maxt=1654214400000 ulid=01G4KYYWQKRNYQRPEK227CM2H3
ts=2022-06-13T14:50:54.113Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1654214408938 maxt=1654279200000 ulid=01G4NWRDWTR71RC67TQDM3W285
ts=2022-06-13T14:50:54.113Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1654279205501 maxt=1654344000000 ulid=01G4QTHYXVG2NCFAQQBXKYFTTV
ts=2022-06-13T14:50:54.113Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1654344005501 maxt=1654408800000 ulid=01G4SRBH4EZW9B3NAJ635FWJRR
ts=2022-06-13T14:50:54.113Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1654408805501 maxt=1654473600000 ulid=01G4VP51EBJVEBETVG1DQ4QJMP
ts=2022-06-13T14:50:54.113Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1654473608938 maxt=1654538400000 ulid=01G4XKYKQFA4RSZEFS3Q8ZTCJH
ts=2022-06-13T14:50:54.113Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1654538405501 maxt=1654603200000 ulid=01G4ZHR499HZ6QWSXPVK7Q4VC7
ts=2022-06-13T14:50:54.113Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1654603205501 maxt=1654668000000 ulid=01G51FHQCFQFDZGVQF6MGGKKGN
ts=2022-06-13T14:50:54.113Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1654668005501 maxt=1654732800000 ulid=01G53DB6P9BYT20D4PAN85F4ZR
ts=2022-06-13T14:50:54.113Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1654732805501 maxt=1654797600000 ulid=01G55B4RBK5R6V0JMX0PHNJFDE
ts=2022-06-13T14:50:54.113Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1654797608938 maxt=1654862400000 ulid=01G578Y8WZ62Z288CRWTTK2S0J
ts=2022-06-13T14:50:54.113Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1654862405501 maxt=1654927200000 ulid=01G596QVAB4Z7CVXD97JWZ7N2F
ts=2022-06-13T14:50:54.113Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1654927208938 maxt=1654992000000 ulid=01G5B4HBPM56WYTNYBGKNM749P
ts=2022-06-13T14:50:54.113Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false
ts=2022-06-13T14:50:54.113Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1654992008938 maxt=1655056800000 ulid=01G5D2AXT9ZFTBFD5PG5H4H176
ts=2022-06-13T14:50:54.113Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1655056808938 maxt=1655078400000 ulid=01G5DPY0H3GRQ0WRYV9CD41KCA
ts=2022-06-13T14:50:54.113Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1655100005501 maxt=1655107200000 ulid=01G5E4NDBNK41MF7KK00ADZZKS
ts=2022-06-13T14:50:54.114Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1655107208938 maxt=1655114400000 ulid=01G5EBH5C4NVRAW1HB0J5BJEJS
ts=2022-06-13T14:50:54.114Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1655078408938 maxt=1655100000000 ulid=01G5EBH6Z15G7JHSCHVCHEMRV5
ts=2022-06-13T14:50:54.114Z caller=repair.go:57 level=info component=tsdb msg="Found healthy block" mint=1655114405501 maxt=1655121600000 ulid=01G5EJCVVMSX9HVGJSZCB4V0CV
ts=2022-06-13T14:50:54.114Z caller=dir_locker.go:77 level=warn component=tsdb msg="A lockfile from a previous execution already existed. It was replaced" file=/data/lock
ts=2022-06-13T14:50:54.114Z caller=main.go:799 level=info msg="Stopping scrape discovery manager..."
ts=2022-06-13T14:50:54.114Z caller=main.go:813 level=info msg="Stopping notify discovery manager..."
ts=2022-06-13T14:50:54.114Z caller=main.go:835 level=info msg="Stopping scrape manager..."
ts=2022-06-13T14:50:54.114Z caller=main.go:809 level=info msg="Notify discovery manager stopped"
ts=2022-06-13T14:50:54.114Z caller=main.go:795 level=info msg="Scrape discovery manager stopped"
ts=2022-06-13T14:50:54.114Z caller=manager.go:946 level=info component="rule manager" msg="Stopping rule manager..."
ts=2022-06-13T14:50:54.114Z caller=manager.go:956 level=info component="rule manager" msg="Rule manager stopped"
ts=2022-06-13T14:50:54.114Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..."
ts=2022-06-13T14:50:54.114Z caller=main.go:1068 level=info msg="Notifier manager stopped"
ts=2022-06-13T14:50:54.114Z caller=main.go:829 level=info msg="Scrape manager stopped"
ts=2022-06-13T14:50:54.114Z caller=main.go:1077 level=error err="opening storage failed: lock DB directory: resource temporarily unavailable"

Answer 1

嗯，我已经通过几个步骤克服了这种情况。我有一个假设，旧的未终止 pod 使用 tsdb。因此，我将前一个 pod 的副本集更改为 0 个副本，新的 pod 正确启动。

Answer 2

是的，我也终止了旧的 Pod，它也对我有用。