Elastic Beanstalk根据4xx响应禁用健康状况更改

问题描述 投票:48回答:7

我在Elastic Beanstalk上运行了一个休息API,效果很好。应用程序方面的一切都运行良好,并按预期工作。

该应用程序是一个rest api,用于查找不同的用户。

example url: http://service.com/user?uid=xxxx&anotherid=xxxx

如果找到具有任一id的用户,则api以200 OK响应,如果没有,则按照404 Not Found响应。 HTTP/1.1状态代码捍卫。

我们的api在很多要求上回答404 Not Found的情况并不少见,而弹性豆茎将我们的环境从OK转移到Warning,甚至转移到Degraded。看起来nginx因为这种退化状态而拒绝连接应用程序。 (看起来warning和Qazxswpoi状态的阈值为30%+ degradeda和50%+。这是一个问题,因为应用程序实际上按预期工作,但Elastic Beanstalks默认设置认为它是一个问题,当它真的没有。

有没有人知道在EB中编辑4xx警告和状态转换的阈值的方法,还是完全禁用它们?

或者我应该真的做一个症状治疗,并停止使用404 Not Found这样的电话? (我真的不喜欢这个选项)

rest amazon-web-services elastic-beanstalk
7个回答
56
投票

更新:AWS EB最终包含一个内置设置:https://stackoverflow.com/a/51556599/1123355

旧解决方案:潜入EB实例并花费几个小时寻找EB的健康检查守护程序实际上将状态代码报告给EB进行评估的地方,我终于找到了它,并提出了一个可以作为一个完美的解决方法的补丁防止4xx响应代码将环境转变为Degraded环境健康状态,以及毫无意义地通过此电子邮件通知您:

Environment health has transitioned from Ok to Degraded. 59.2 % of the requests are erroring with HTTP 4xx.

状态代码报告逻辑位于healthd-appstat中,这是由EB团队开发的Ruby脚本,它不断监视/var/log/nginx/access.log并将状态代码报告给EB,具体在以下路径中:

/opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/healthd-appstat-1.0.1/lib/healthd-appstat/plugin.rb

以下.ebextensions文件将修补此Ruby脚本,以避免将4xx响应代码报告回EB。这意味着由于4xx错误,EB永远不会降低环境健康状况,因为它只是不知道它们正在发生。这也意味着EB环境中的“运行状况”页面将始终显示0响应代码计数的4xx

container_commands:
    01-patch-healthd:
        command: "sudo /bin/sed -i 's/\\# normalize units to seconds with millisecond resolution/if status \\&\\& status.index(\"4\") == 0 then next end/g' /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/healthd-appstat-1.0.1/lib/healthd-appstat/plugin.rb"
    02-restart-healthd:
        command: "sudo /usr/bin/kill $(/bin/ps aux | /bin/grep -e '/bin/bash -c healthd' | /usr/bin/awk '{ print $2 }')"
        ignoreErrors: true

是的,它有点难看,但它完成了工作,至少在EB团队通过一些配置参数提供忽略4xx错误的方法之前。在部署时,将其与您的应用程序一起包含在相对于项目根目录的以下路径中:

.ebextensions/ignore_4xx.config

祝你好运,如果有帮助,请告诉我!


21
投票

有一个名为Ignore HTTP 4xx的专用运行状况监控规则自定义(附带屏幕截图)只需启用它,EB就不会因4xx错误而降低实例运行状况。 enter image description here


14
投票

谢谢你的回答Elad Nava,我有同样的问题,你的解决方案对我来说非常合适!

但是,在AWS Support Center中打开票证后,他们建议我修改nginx配置以忽略运行状况检查中的4xx而不是修改ruby脚本。为此,我还必须在.ebextensions目录中添加一个配置文件,以覆盖默认的nginx.conf文件:

files:
  "/tmp/nginx.conf":
    content: |

      # Elastic Beanstalk Managed

      # Elastic Beanstalk managed configuration file
      # Some configuration of nginx can be by placing files in /etc/nginx/conf.d
      # using Configuration Files.
      # http://docs.amazonwebservices.com/elasticbeanstalk/latest/dg/customize-containers.html
      #
      # Modifications of nginx.conf can be performed using container_commands to modify the staged version
      # located in /tmp/deployment/config/etc#nginx#nginx.conf

      # Elastic_Beanstalk
      # For more information on configuration, see:
      #   * Official English Documentation: http://nginx.org/en/docs/
      #   * Official Russian Documentation: http://nginx.org/ru/docs/

      user  nginx;
      worker_processes  auto;

      error_log  /var/log/nginx/error.log;

      pid        /var/run/nginx.pid;


      worker_rlimit_nofile 1024;

      events {
          worker_connections  1024;
      }

      http {

          ###############################
          # CUSTOM CONFIG TO IGNORE 4xx #
          ###############################

          map $status $loggable {
            ~^[4]  0;
            default 1;
          }

          map $status $modstatus {
            ~^[4]  200;
            default $status;
          }

          #####################
          # END CUSTOM CONFIG #
          #####################

          port_in_redirect off;
          include       /etc/nginx/mime.types;
          default_type  application/octet-stream;


          # This log format was modified to ignore 4xx status codes!
          log_format  main   '$remote_addr - $remote_user [$time_local] "$request" '
                             '$status $body_bytes_sent "$http_referer" '
                             '"$http_user_agent" "$http_x_forwarded_for"';

          access_log  /var/log/nginx/access.log  main;

          log_format healthd '$msec"$uri"'
                             '$modstatus"$request_time"$upstream_response_time"'
                             '$http_x_forwarded_for' if=$loggable;

          sendfile        on;
          include /etc/nginx/conf.d/*.conf;

          keepalive_timeout  1200;

      }

container_commands:
  01_modify_nginx:
    command: cp /tmp/nginx.conf /tmp/deployment/config/#etc#nginx#nginx.conf

虽然这个解决方案非常冗长,但我个人认为实施起来更安全,只要它不依赖于任何AWS专有脚本。我的意思是,如果出于某种原因,AWS决定删除或修改他们的ruby脚本(相信我或不相信,他们喜欢在没有事先通知的情况下更改脚本),很有可能使用sed的解决方法不再适用。


2
投票

这是一个基于Adriano Valente's answer的解决方案。我无法让$loggable位工作,虽然跳过404的记录似乎是一个很好的解决方案。我只是创建了一个新的.conf文件,定义了$modstatus变量,然后覆盖了healthd日志格式,使用$modstatus代替$status。此更改还需要重新启动nginx。这适用于运行Ruby 2.3(Puma)的Elastic Beanstalk的64位Amazon Linux 2016.09 v2.3.1。

# .ebextensions/nginx.conf

files:
  "/tmp/nginx.conf":
    content: |

      # Custom config to ignore 4xx in the health file only
      map $status $modstatus {
        ~^[4]  200;
        default $status;
      }

container_commands:
  modify_nginx_1:
    command: "cp /tmp/nginx.conf /etc/nginx/conf.d/custom_status.conf"
  modify_nginx_2:
    command: sudo sed -r -i 's@\$status@$modstatus@' /opt/elasticbeanstalk/support/conf/webapp_healthd.conf
  modify_nginx_3:
    command: sudo /etc/init.d/nginx restart

1
投票

基于Elad Nava's Answer,我认为最好直接使用elasticbeanstalk healthd的控制脚本而不是kill:

container_commands:
    01-patch-healthd:
        command: "sudo /bin/sed -i 's/\\# normalize units to seconds with millisecond resolution/if status \\&\\& status.index(\"4\") == 0 then next end/g' /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/healthd-appstat-1.0.1/lib/healthd-appstat/plugin.rb"
    02-restart-healthd:
        command: "sudo /opt/elasticbeanstalk/bin/healthd-restart"

最后,在调查此问题时,我注意到健康和apache日志状态代码与使用%s的前者不同,而后者%> s导致它们之间存在差异。我还修补了这个:

    03-healthd-logs:
        command: sed -i 's/^LogFormat.*/LogFormat "%{%s}t\\"%U\\"%>s\\"%D\\"%D\\"%{X-Forwarded-For}i" healthd/g' /etc/httpd/conf.d/healthd.conf

1
投票

自2018年4月起AWS支持提供的解决方案:

 files:
  "/tmp/custom-site-nginx.conf":
    mode: "000664"
    owner: root
    group: root
    content: |
       map $http_upgrade $connection_upgrade {
           default        "upgrade";
           ""            "";
       }
       # Elastic Beanstalk Modification(EB_INCLUDE)
       # Custom config
       # HTTP 4xx ignored.
       map $status $loggable {
                ~^[4]  0;
                default 1;
       }


       server {
           listen 80;

         gzip on;
         gzip_comp_level 4;
         gzip_types text/html text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

           if ($time_iso8601 ~ "^(\d{4})-(\d{2})-(\d{2})T(\d{2})") {
               set $year $1;
               set $month $2;
               set $day $3;
               set $hour $4;
           }
           access_log /var/log/nginx/healthd/application.log.$year-$month-$day-$hour healthd if=$loggable;

           access_log    /var/log/nginx/access.log;

           location / {
               proxy_pass            http://docker;
               proxy_http_version    1.1;

               proxy_set_header    Connection            $connection_upgrade;
               proxy_set_header    Upgrade                $http_upgrade;
               proxy_set_header    Host                $host;
               proxy_set_header    X-Real-IP            $remote_addr;
               proxy_set_header    X-Forwarded-For        $proxy_add_x_forwarded_for;
           }
       }

 container_commands:
   override_beanstalk_nginx:
     command: "mv -f /tmp/custom-site-nginx.conf /etc/nginx/sites-available/elasticbeanstalk-nginx-docker-proxy.conf"

1
投票

我最近遇到了同样被4xx错误轰炸的问题。我尝试了上面列出的建议,但没有任何对我有用。我联系了AWS Support,这是他们的建议,它解决了我的问题。我有一个运行2个实例的Elastic Beanstalk应用程序。

  1. 创建一个名为.ebextensions的文件夹
  2. 在这个文件夹中,创建一个名为nginx.config的文件(确保它具有.config扩展名。“。conf”不会这样做!)
  3. 如果要使用Docker容器部署应用程序,请确保部署包中包含此.ebextensions文件夹。对我来说,捆绑包包括文件夹以及Dockerrun.aws.json

以下是nginx.config文件的全部内容:

files:
  "/etc/nginx/nginx.conf":
    content: |
      # Elastic Beanstalk Nginx Configuration File
      user  nginx;
      worker_processes  auto;

      error_log  /var/log/nginx/error.log;

      pid        /var/run/nginx.pid;

      events {
          worker_connections  1024;
      }

      http {

          # Custom config
          # HTTP 4xx ignored.
          map $status $loggable {
            ~^[4]  0;
            default 1;
          }

          # Custom config
          # HTTP 4xx ignored.
          map $status $modstatus {
            ~^[4]  200;
            default $status;
          }

          include       /etc/nginx/mime.types;
          default_type  application/octet-stream;

          access_log    /var/log/nginx/access.log;

          log_format  healthd '$msec"$uri"$modstatus"$request_time"$upstream_response_time"$http_x_forwarded_for';

          include       /etc/nginx/conf.d/*.conf;
          include       /etc/nginx/sites-enabled/*;
      }
© www.soinside.com 2019 - 2024. All rights reserved.