RestTemplate + ConnectionPoolTimeoutException: Timeout waiting for connection from pool

Question

I got this error all of a sudden in production while the application was not under any load.

The issue happened when my code tries to send the PUT message using spri

Answer 1

Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool

This error is self describing. You need to increase your connection pool in production - current implementation of HttpComponentsClientHttpRequestFactory default constructor is using HttpClientBuilder with .useSystemProperties().

I believe it will be 5 connections by default. This works for client but is unlikely what you want in server environment. You need to use something like

new RestTemplate(new HttpComponentsClientHttpRequestFactory(HttpClientBuilder.create()
                    .setMaxConnTotal(200)
                    .setMaxConnPerRoute(50)
                    .build()));

Answer 2

The problem is the HTTP client connections aren't being closed. I had the same problem with a service that only has about 1 request per second.

"exception":"org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool"

You need to add a finally block and close the connection.

Answer 3

I would suggest to configure HttpComponentsClientHttpRequestFactory instance being passed in the constructor of RestTemplate increasing defaultMaxPerRoute or maxPerRoute for the specific http route for which requests are timing out, increasing the pool size is not enough, as I mentioned in a comment, even if you set PoolingHttpClientConnectionManager.setMaxTotal() to 200, HttpComponentsClientHttpRequestFactory uses a defaultMaxPerRoute of 4, my guess would be in an attempt for a host route (scheme, host, port) not to hijack the connection pool)

...
public PoolingHttpClientConnectionManager poolingHttpClientConnectionManager() {
        PoolingHttpClientConnectionManager result = new PoolingHttpClientConnectionManager();
        result.setMaxTotal(this.httpHostConfiguration.getMaxTotal());
        // Default max per route is used in case it's not set for a specific route
        result.setDefaultMaxPerRoute(this.httpHostConfiguration.getDefaultMaxPerRoute());
        // and / or
        if (CollectionUtils.isNotEmpty(this.httpHostConfiguration.getMaxPerRoutes())) {
            for (HttpHostConfiguration httpHostConfig : this.httpHostConfiguration.getMaxPerRoutes()) {
                HttpHost host = new HttpHost(httpHostConfig.getHost(), httpHostConfig.getPort(), httpHostConfig.getScheme());
                // Max per route for a specific host route
                result.setMaxPerRoute(new HttpRoute(host), httpHostConfig.getMaxPerRoute());
            }
        }
        return result;
    }
  ...


@Configuration
@ConfigurationProperties(prefix = "httpConnPool")
public class HttpHostsConfiguration {

  private Integer maxTotal;
  private Integer defaultMaxPerRoute;
  private List<HttpHostConfiguration> maxPerRoutes;

  // Getters, Setters
...

application.yml

httpConnPool:
  maxTotal: 20
  defaultMaxPerRoute: 20
  maxPerRoutes:
    -
      scheme: http
      host: localhost
      port: 8800
      maxPerRoute: 20

I recently blog about Troubleshooting Spring's RestTemplate Requests Timeout where requests timing out were troubleshooted using JMeter and shell commands and fixed via configuration settings.