为什么我对 nova-api 的请求收到错误代码 500“socket.timeout”?

问题描述 投票:0回答:0

在大规模场景中,我们可能会增加worker的数量或nova-api的数量来提高最大请求处理能力。虽然worker数量没有必要达到30人,但是当我将worker数量增加到30人甚至更多时,出现了一些错误。

在我的测试中,当并发请求量达到200-300时,就会出现下面的错误。但是随着并发请求量的增加,错误就会消失。

"message": "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\n<class 'socket.timeout'>", "code": 500}

OpenStack 版本为 Train。

我查看了nova-api的日志,日志是这样的:

2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi [req-4c1dc874-db6b-4bae-99c7-f0a73584d850 0966b6f07f964abb8cfc43ac36103ce5 b2117415705d4d18b7d6c1458e8818c6 - default default] Unexpected exception in API method: timeout: timed out
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 671, in wrapped
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/api/validation/init.py", line 192, in wrapper
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/api/validation/init.py", line 192, in wrapper
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/api/validation/init.py", line 192, in wrapper
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/api/validation/init.py", line 192, in wrapper
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/api/validation/init.py", line 192, in wrapper
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 143, in detail
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi     servers = self._get_servers(req, is_detail=True)
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 344, in _get_servers
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi     req, instance_list, cell_down_support=cell_down_support)
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/api/openstack/compute/views/servers.py", line 403, in detail
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi     cell_down_support=cell_down_support)
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/api/openstack/compute/views/servers.py", line 446, in _list_view
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi     if server.uuid != virtual_interface.FAKE_UUID]
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/api/openstack/compute/views/servers.py", line 253, in show
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi     az = avail_zone.get_instance_availability_zone(context, instance)
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/availability_zones.py", line 200, in get_instance_availability_zone
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi     cache.set(cache_key, az)
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/cache_utils.py", line 116, in set
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi     return self.region.set(key, value)
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/dogpile/cache/region.py", line 1030, in set
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi     self.backend.set(key, self._value(value))
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/dogpile/cache/backends/memcached.py", line 178, in set
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi     **self.set_arguments
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_cache/backends/memcache_pool.py", line 32, in _run_method
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi     return getattr(client, __name)(*args, **kwargs)
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/memcache.py", line 727, in set
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi     return self._set("set", key, val, time, min_compress_len, noreply)
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/memcache.py", line 1016, in _set
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi     server, key = self._get_server(key)
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/memcache.py", line 433, in _get_server
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi     if server.connect():
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/memcache.py", line 1381, in connect
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi     if self._get_socket():
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/memcache.py", line 1413, in _get_socket
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi     self.flush()
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/memcache.py", line 1488, in flush
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi     self.expect(b'OK')
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/memcache.py", line 1463, in expect
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi     line = self.readline(raise_exception)
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/memcache.py", line 1449, in readline
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi     data = recv(4096)
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/greenio/base.py", line 366, in recv
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi     return self._recv_loop(self.fd.recv, b'', bufsize, flags)
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/greenio/base.py", line 360, in _recv_loop
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi     self._read_trampoline()
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/greenio/base.py", line 331, in _read_trampoline
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi     timeout_exc=socket_timeout('timed out'))
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/greenio/base.py", line 210, in _trampoline
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi     mark_as_closed=self._mark_as_closed)
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/hubs/init.py", line 159, in trampoline
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi     return hub.switch()
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi   File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 298, in switch
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi     return self.greenlet.switch()
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi timeout: timed out
2023-08-01 10:31:47.472 42 ERROR nova.api.openstack.wsgi
openstack openstack-nova
© www.soinside.com 2019 - 2024. All rights reserved.