获取java.net.SocketTimeoutException：读取超时但tcpdump为空

Question

我使用HttpRestTemplate来调用另一台服务器。一段时间后，集群中的所有服务都会出现Java Read time out错误。没有显示任何内容（只是空）。重启后一切都变好了。这种奇怪行为的原因是什么？

我使用apache http client 4.5.1 Java 1.8.152 spring web 4.2.2

HttpRestTemplate配置：

HTTP://呜呜呜.spring framework.org/schema/beans/spring-beans.下水道">

<bean id="recomenderRestTemplate" class="org.springframework.web.client.RestTemplate">
    <constructor-arg>
        <bean class="org.springframework.http.client.HttpComponentsClientHttpRequestFactory">
            <property name="connectTimeout" value="300"/>
            <property name="readTimeout" value="400"/>
            <property name="connectionRequestTimeout" value="100"/>
            <property name="httpClient">
                <bean factory-bean="recomenderHttpClientBuilder" factory-method="build"/>
            </property>
        </bean>
    </constructor-arg>
</bean>

<bean id="recomenderHttpClientBuilder" class="org.apache.http.impl.client.HttpClients" factory-method="custom">
    <property name="connectionManager">
        <bean class="org.apache.http.impl.conn.PoolingHttpClientConnectionManager">
            <constructor-arg name="timeToLive" value="3600"/>
            <constructor-arg>
                <value type="java.util.concurrent.TimeUnit">SECONDS</value>
            </constructor-arg>
            <property name="maxTotal" value="300"/>
            <property name="defaultMaxPerRoute" value="300"/>
        </bean>
    </property>
    <property name="requestExecutor">
        <bean class="ru.yandex.kino.utils.profiling.ProfilingHttpRequestExecutor">
            <constructor-arg name="infoLimit" value="0"/>
            <constructor-arg name="warnLimit" value="100"/>
            <constructor-arg name="errorLimit" value="300"/>
        </bean>
    </property>
</bean>

错误

ERROR 2018-07-23 12:39:46,412 r.y.k.t.a.s.RecomenderService - Recommendation call for region=39, lat=47.222531, lon=47.222531 ses456639
org.springframework.web.client.ResourceAccessException: I/O error on POST request for "http://mediapers-tv.yandex.net/recommender?client=clientt":Read timed out; nested exception is java.net.SocketTimeoutException: Read timed out
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:607)
    at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:572)
    at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:493)
    at ru.yandex.kino.tv.api.client.recomender.RecomenderClient.rank(RecomenderClient.java:78)
    at ru.yandex.kino.tv.api.client.recomender.RecomenderClient$$FastClassBySpringCGLIB$$9c6b4439.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:717)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
    at ru.yandex.kino.utils.monitoring.MonitoringTemplate.monitor(MonitoringTemplate.java:7)
    at ru.yandex.kino.utils.monitoring.interceptor.MonitoringInterceptor.invoke(MonitoringInterceptor.java:29)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at ru.yandex.kino.utils.monitoring.MonitoringTemplate.monitor(MonitoringTemplate.java:7)
    at ru.yandex.kino.utils.monitoring.interceptor.MonitoringInterceptor.invoke(MonitoringInterceptor.java:29)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at ru.yandex.kino.utils.monitoring.MonitoringTemplate.monitor(MonitoringTemplate.java:7)
    at ru.yandex.kino.utils.monitoring.interceptor.MonitoringInterceptor.invoke(MonitoringInterceptor.java:29)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653)
    at ru.yandex.kino.tv.api.client.recomender.RecomenderClient$$EnhancerBySpringCGLIB$$9ff58d5.rank(<generated>)
    at ru.yandex.kino.tv.api.service.RecomenderService.rank(RecomenderService.java:49)
    at ru.yandex.kino.tv.api.service.RecomenderService$$FastClassBySpringCGLIB$$bdf0f1df.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:717)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
    at org.springframework.retry.interceptor.StatefulRetryOperationsInterceptor$MethodInvocationRetryCallback.doWithRetry(StatefulRetryOperationsInterceptor.java:229)
    at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:287)
    at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:211)
    at org.springframework.retry.interceptor.StatefulRetryOperationsInterceptor.invoke(StatefulRetryOperationsInterceptor.java:173)
    at org.springframework.retry.annotation.AnnotationAwareRetryOperationsInterceptor.invoke(AnnotationAwareRetryOperationsInterceptor.java:153)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653)
    at ru.yandex.kino.tv.api.service.RecomenderService$$EnhancerBySpringCGLIB$$be59ec2b.rank(<generated>)
    at ru.yandex.kino.tv.api.web.controllers.OnAirController.getRecommendedOnAirList(OnAirController.java:194)
    at ru.yandex.kino.tv.api.web.controllers.OnAirController$$FastClassBySpringCGLIB$$bc4c1493.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:717)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
    at org.springframework.validation.beanvalidation.MethodValidationInterceptor.invoke(MethodValidationInterceptor.java:139)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653)
    at ru.yandex.kino.tv.api.web.controllers.OnAirController$$EnhancerBySpringCGLIB$$9adc6d16.getRecommendedOnAirList(<generated>)
    at sun.reflect.GeneratedMethodAccessor385.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:137)
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:806)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:729)
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:735)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:648)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1336)
    at ru.yandex.kino.utils.graphite.MetricsFilter.doFilterInternal(MetricsFilter.java:32)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
    at ru.yandex.kino.utils.web.TimeRoundingFilter.doFilter(TimeRoundingFilter.java:57)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:85)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
    at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:154)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    at ru.yandex.kino.tv.api.web.Uid2MdcFilter.doFilter(Uid2MdcFilter.java:28)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    at ru.yandex.kino.tv.api.web.security.AuthVersionFilter.doFilter(AuthVersionFilter.java:27)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    at ru.yandex.kino.utils.blackbox.spring.OAuthAuthenticationFilter.doFilter(OAuthAuthenticationFilter.java:49)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    at ru.yandex.kino.utils.access.AccessRestrictionFilter.doFilter(AccessRestrictionFilter.java:42)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    at ru.yandex.kino.tv.api.web.security.AuthCratchParameterFilter.doFilter(AuthCratchParameterFilter.java:40)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    at ru.yandex.kino.tv.api.web.security.AuthCratchFilter.doFilter(AuthCratchFilter.java:68)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    at ru.yandex.kino.utils.healthcheck.ResponseHealthCheckFilter.doFilterInternal(ResponseHealthCheckFilter.java:54)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
    at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
    at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
    at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:559)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1072)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
    at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
    at org.eclipse.jetty.server.Server.handle(Server.java:365)
    at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485)
    at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:937)
    at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:998)
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:856)
    at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
    at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:627)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:51)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
    at java.net.SocketInputStream.read(SocketInputStream.java:171)
    at java.net.SocketInputStream.read(SocketInputStream.java:141)
    at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
    at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155)
    at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284)
    at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
    at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
    at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
    at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
    at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
    at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
    at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
    at ru.yandex.kino.utils.profiling.ProfilingHttpRequestExecutor.execute(ProfilingHttpRequestExecutor.java:52)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
    at org.springframework.http.client.HttpComponentsClientHttpRequest.executeInternal(HttpComponentsClientHttpRequest.java:91)
    at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48)
    at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53)
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:596)
    ... 112 more 

Answer 1

尝试增加下面代码中定义的读取超时值

<bean id="recomenderRestTemplate" class="org.springframework.web.client.RestTemplate">
    <constructor-arg>
        <bean class="org.springframework.http.client.HttpComponentsClientHttpRequestFactory">
            <property name="connectTimeout" value="300"/>
            <property name="readTimeout" value="400"/>
            <property name="connectionRequestTimeout" value="100"/>
            <property name="httpClient">
                <bean factory-bean="recomenderHttpClientBuilder" factory-method="build"/>
            </property>
        </bean>
    </constructor-arg>
</bean>

HttpComponentsClientHttpRequestFactory以毫秒为单位超时。

访问spring java docs的春季官方文档

此外，根据堆栈跟踪，很明显它是一个读取超时的情况

org.springframework.web.client.ResourceAccessException: I/O error on POST request for "http://mediapers-tv.yandex.net/recommender?client=clientt":Read timed out; nested exception is java.net.SocketTimeoutException: Read timed out

Answer 2

https://stackoverflow.com/a/55059992/908827看到我的答案。读取输入流时，此消息会被警告，没有数据发送到网络层。