我目前在使用 varnish 7.0.3 时遇到问题
我在 Magento 前面使用清漆。现在我遇到了清漆瞬态存储内存泄漏的问题。
从我在互联网上看到的，libjemalloc2 版本“libjemalloc2/jammy，现在 5.2.1-4ubuntu1”对此有一个问题，他们说 5.3.0 解决了这个问题。
如何将libjemalloc2升级到5.3.0？我使用的是 Ubuntu 22.04，所有内容都是使用 apt 包管理器安装的。
Varnish 核心开发人员小组不知道有关 jemalloc 的任何实际内存泄漏。您能描述一下这次泄漏的症状吗？
瞬态存储是Varnish中非常特殊的存储引擎。默认情况下，它是无限制的，并确保客户端处理时保存短暂且不可缓存的内容。
临时存储内存的增长速度比收缩速度快得多，潜在的 jemalloc 调整可以优化一些浪费的内存。
您有什么特别可以分享的来支持您的问题吗？因为它完全有可能是一个功能而不是一个错误。
请检查 Varnish 6 LTS /w CentOS 8 不考虑内存限制？
我也遇到同样的情况。我正在使用具有 jemalloc 5.2.1 的 ubuntu 22.04 LTS .
varnish.service 现在看起来像这样：
[Service]
Type=forking
KillMode=process
Environment="MALLOC_CONF=thp:never,narenas:2"
# Maximum number of open files (for ulimit -n)
LimitNOFILE=131072
# Locked shared memory - should suffice to lock the shared memory log
# (varnishd -l argument)
# Default log size is 80MB vsl + 1M vsm + header -> 82MB
# unit is bytes
LimitMEMLOCK=85983232
# Enable this to avoid "fork failed" on reload.
TasksMax=infinity
# Maximum size of the corefile.
LimitCORE=infinity
ExecStart=/usr/sbin/varnishd \
-a :80 \
-p feature=+http2 \
-p feature=+esi_ignore_other_elements \
-p feature=+esi_disable_xml_check \
-p feature=+esi_ignore_https \
-p http_req_size=256k \
-p http_resp_hdr_len=128k \
-p http_resp_size=512k \
-p workspace_backend=512k \
-p workspace_client=256k \
-f /etc/varnish/default.vcl \
-s Cache=malloc,2g \
-s Transient=malloc,512m
ExecReload=/usr/sbin/varnishreload
[Install]
WantedBy=multi-user.target
我的 vcl 看起来像这样：
vcl 4.0;
import std;
# The minimal Varnish version is 6.0
# For SSL offloading, pass the following header in your proxy server or load balancer: 'X-Forwarded-Proto: https'
backend default {
.host = "xxx.xx.xx.xx";
.port = "80";
.first_byte_timeout = 30s;
.probe = {
.url = "/health_check.php";
.timeout = 2s;
.interval = 2m;
.window = 10;
.threshold = 5;
}
}
acl purge {
"xx.xxx.xx.xx";
}
sub vcl_recv {
if (req.restarts > 0) {
set req.hash_always_miss = true;
}
if (req.method == "PURGE") {
if (client.ip !~ purge) {
return (synth(405, "Method not allowed"));
}
# To use the X-Pool header for purging varnish during automated deployments, make sure the X-Pool header
# has been added to the response in your backend server config. This is used, for example, by the
# capistrano-magento2 gem for purging old content from varnish during it's deploy routine.
if (!req.http.X-Magento-Tags-Pattern && !req.http.X-Pool) {
return (synth(400, "X-Magento-Tags-Pattern or X-Pool header required"));
}
if (req.http.X-Magento-Tags-Pattern) {
ban("obj.http.X-Magento-Tags ~ " + req.http.X-Magento-Tags-Pattern);
}
if (req.http.X-Pool) {
ban("obj.http.X-Pool ~ " + req.http.X-Pool);
}
return (synth(200, "Purged"));
}
if (req.method != "GET" &&
req.method != "HEAD" &&
req.method != "PUT" &&
req.method != "POST" &&
req.method != "TRACE" &&
req.method != "OPTIONS" &&
req.method != "DELETE") {
/* Non-RFC2616 or CONNECT which is weird. */
return (pipe);
}
# We only deal with GET and HEAD by default
if (req.method != "GET" && req.method != "HEAD") {
return (pass);
}
# Bypass customer, shopping cart, checkout
if (req.url ~ "/customer" || req.url ~ "/checkout") {
return (pass);
}
# Bypass health check requests
if (req.url ~ "^/(pub/)?(health_check.php)$") {
return (pass);
}
# Set initial grace period usage status
set req.http.grace = "none";
# normalize url in case of leading HTTP scheme and domain
set req.url = regsub(req.url, "^http[s]?://", "");
# collect all cookies
std.collect(req.http.Cookie);
# Compression filter. See https://www.varnish-cache.org/trac/wiki/FAQ/Compression
if (req.http.Accept-Encoding) {
if (req.url ~ "\.(jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf|flv)$") {
# No point in compressing these
unset req.http.Accept-Encoding;
} elsif (req.http.Accept-Encoding ~ "gzip") {
set req.http.Accept-Encoding = "gzip";
} elsif (req.http.Accept-Encoding ~ "deflate" && req.http.user-agent !~ "MSIE") {
set req.http.Accept-Encoding = "deflate";
} else {
# unknown algorithm
unset req.http.Accept-Encoding;
}
}
# Remove all marketing get parameters to minimize the cache objects
if (req.url ~ "(\?|&)(gclid|cx|ie|cof|siteurl|zanpid|origin|fbclid|mc_[a-z]+|utm_[a-z]+|_bta_[a-z]+)=") {
set req.url = regsuball(req.url, "(gclid|cx|ie|cof|siteurl|zanpid|origin|fbclid|mc_[a-z]+|utm_[a-z]+|_bta_[a-z]+)=[-_A-z0-9+()%.]+&?", "");
set req.url = regsub(req.url, "[?|&]+$", "");
}
# Static files caching
if (req.url ~ "^/(pub/)?(media|static)/") {
# Static files should not be cached by default
return (pass);
# But if you use a few locales and don't use CDN you can enable caching static files by commenting previous line (#return (pass);) and uncommenting next 3 lines
#unset req.http.Https;
#unset req.http.X-Forwarded-Proto;
#unset req.http.Cookie;
}
# Bypass authenticated GraphQL requests without a X-Magento-Cache-Id
if (req.url ~ "/graphql" && !req.http.X-Magento-Cache-Id && req.http.Authorization ~ "^Bearer") {
return (pass);
}
return (hash);
}
sub vcl_hash {
if ((req.url !~ "/graphql" || !req.http.X-Magento-Cache-Id) && req.http.cookie ~ "X-Magento-Vary=") {
hash_data(regsub(req.http.cookie, "^.*?X-Magento-Vary=([^;]+);*.*$", "\1"));
}
# To make sure http users don't see ssl warning
if (req.http.X-Forwarded-Proto) {
hash_data(req.http.X-Forwarded-Proto);
}
if (req.url ~ "/graphql") {
call process_graphql_headers;
}
}
sub process_graphql_headers {
if (req.http.X-Magento-Cache-Id) {
hash_data(req.http.X-Magento-Cache-Id);
# When the frontend stops sending the auth token, make sure users stop getting results cached for logged-in users
if (req.http.Authorization ~ "^Bearer") {
hash_data("Authorized");
}
}
if (req.http.Store) {
hash_data(req.http.Store);
}
if (req.http.Content-Currency) {
hash_data(req.http.Content-Currency);
}
}
sub vcl_backend_response {
set beresp.grace = 3d;
if (beresp.http.content-type ~ "text") {
set beresp.do_esi = true;
}
if (bereq.url ~ "\.js$" || beresp.http.content-type ~ "text") {
set beresp.do_gzip = true;
}
if (beresp.http.X-Magento-Debug) {
set beresp.http.X-Magento-Cache-Control = beresp.http.Cache-Control;
}
# cache only successfully responses and 404s that are not marked as private
if (beresp.status != 200 &&
beresp.status != 404 &&
beresp.http.Cache-Control ~ "private") {
set beresp.uncacheable = true;
set beresp.ttl = 86400s;
return (deliver);
}
# validate if we need to cache it and prevent from setting cookie
if (beresp.ttl > 0s && (bereq.method == "GET" || bereq.method == "HEAD")) {
unset beresp.http.set-cookie;
}
# If page is not cacheable then bypass varnish for 2 minutes as Hit-For-Pass
if (beresp.ttl <= 0s ||
beresp.http.Surrogate-control ~ "no-store" ||
(!beresp.http.Surrogate-Control &&
beresp.http.Cache-Control ~ "no-cache|no-store") ||
beresp.http.Vary == "*") {
# Mark as Hit-For-Pass for the next 2 minutes
set beresp.ttl = 120s;
set beresp.uncacheable = true;
}
# If the cache key in the Magento response doesn't match the one that was sent in the request, don't cache under the request's key
if (bereq.url ~ "/graphql" && bereq.http.X-Magento-Cache-Id && bereq.http.X-Magento-Cache-Id != beresp.http.X-Magento-Cache-Id) {
set beresp.ttl = 0s;
set beresp.uncacheable = true;
}
return (deliver);
}
sub vcl_deliver {
if (resp.http.x-varnish ~ " ") {
set resp.http.X-Magento-Cache-Debug = "HIT";
set resp.http.Grace = req.http.grace;
} else {
set resp.http.X-Magento-Cache-Debug = "MISS";
}
# Not letting browser to cache non-static files.
if (resp.http.Cache-Control !~ "private" && req.url !~ "^/(pub/)?(media|static|repository)/") {
set resp.http.Pragma = "no-cache";
set resp.http.Expires = "-1";
set resp.http.Cache-Control = "no-store, no-cache, must-revalidate, max-age=0";
}
if (!resp.http.X-Magento-Debug) {
unset resp.http.Age;
}
unset resp.http.X-Magento-Debug;
unset resp.http.X-Magento-Tags;
unset resp.http.X-Powered-By;
unset resp.http.Server;
unset resp.http.X-Varnish;
unset resp.http.Via;
unset resp.http.Link;
}
sub vcl_hit {
if (obj.ttl >= 0s) {
# Hit within TTL period
return (deliver);
}
if (std.healthy(req.backend_hint)) {
if (obj.ttl + 300s > 0s) {
# Hit after TTL expiration, but within grace period
set req.http.grace = "normal (healthy server)";
return (deliver);
} else {
# Hit after TTL and grace expiration
return (restart);
}
} else {
# server is not healthy, retrieve from cache
set req.http.grace = "unlimited (unhealthy server)";
return (deliver);
}
}
根据 varnishstat 我当前的存储使用情况：
SMA.Cache.c_req 560519 3.00 5.09 6.84 6.67 6.67
SMA.Cache.c_bytes 5.41G 26.28K 51.53K 68.42K 66.72K 66.72KSMA.Cache.c_freed 5.11G 15.97K 48.68K 52.50K 74.28K 74.28KSMA.Cache.g_alloc 65665 2.00 . 65647.17 65661.34 65661.34
SMA.Cache.g_bytes 307.05M 10.30K . 306.92M 307.09M 307.09MSMA.Cache.g_space 1.70G -10.30K . 1.70G 1.70G 1.70GSMA.Transient.c_req 909978 14.98 8.26 8.48 11.18 11.18
SMA.Transient.c_fail 3579 0.00 0.03 0.56 1.64 1.64
SMA.Transient.c_bytes 4.32G 110.74K 41.16K 57.47K 68.57K 68.57KSMA.Transient.c_freed 3.82G 104.61K 36.40K 54.04K 70.60K 70.60KSMA.Transient.g_alloc 94503 1.00 . 94495.50 94490.31 94490.31
SMA.Transient.g_bytes 511.89M 6.13K . 511.84M 511.81M 511.81MSMA.Transient.g_space 116.70K -6.13K . 161.41K 191.47K 191.47K
根据“top”命令，我的进程现在消耗 1.3g。
问题是我没有看到瞬态内存每次都下降，并且根据 varnishstat，它达到了定义的总瞬态内存的限制。如果我设置一个更大的限制，它会在一天之内达到它并且永远不会下降。此外，清漆过程的总内存最终会大于我对其施加的限制。
有几次它会消耗所有服务器内存并最终使用交换，直到达到极限。
此时操作系统会终止 varnish 进程。