httpurlconnection is not giving proper response code in case of multithreading

问题

I want to get all urls from given url and I want to check status of each url.

For that I am using ExecutorService for multi-threading to check status of each and every url.

Code of class which check response code for url is given below

public class ConnectionTester  implements Callable<Object> {
    private URL url;
    private Map<String,Integer> map;
    private static final Log LOGGER =      
    LogFactoryUtil.getLog(ConnectionTester.class);

    public ConnectionTester(String url,Map<String,Integer> map) {
        try {
            this.url = new URL(url);
            this.map = map;
        } catch (MalformedURLException e) {
            LOGGER.error(e.getMessage(), e);
        }
    }

    /**
     * Get status code of url
     * @return
     */
    public void getStatuscode(URL url) {
        HttpURLConnection http = null;
        try {
            http = (HttpURLConnection)url.openConnection();
            http.setConnectTimeout(0);
            http.setReadTimeout(0);
            http.setRequestMethod("HEAD");
            http.setRequestProperty("User-Agent","Mozilla/5.0 (Windows NT 10.0;    
            Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)   
            Chrome/56.0.2924.87 Safari/537.36");
            http.connect();
            map.put(url.toString(), http.getResponseCode());
        } catch (IOException e) {
            map.put(url.toString(), 500 );
        }finally{
            if(http !=null)http.disconnect();
        }
    }

    @Override
    public Object call() throws Exception {
        getStatuscode(url);
        return null;

    }
}

And I use following code to check response code for each and every url

ExecutorService service = Executors.newFixedThreadPool(urls.size());
   List<ConnectionTester> connectionTesters = new ArrayList<ConnectionTester>(urls.size());
   Map<String,Integer> map  = new HashMap<String, Integer>();
   for (String string : urls) {
       if(Validator.isNotNull(string) && !string.contains("mailto"))
           connectionTesters.add(new ConnectionTester(string, map));
   }
   service.invokeAll(connectionTesters);

Now problem is when I don't use multi-threading I get proper response code of each and every URL but when I use multi-threading I am getting connection timeout exception.

So far I have checked and tried below things.

My internet speed is high
I have set http.setConnectTimeout(0); http.setReadTimeout(0); to set infinite time out.
Response time of url I am checking is also less and working fine.

What am I missing here ?

回答1:

You are overloading the server. When a Unix/Linux TCP server's listen(2) backlog queue fills, it starts ignoring incoming connect requests, which causes connect timeouts at the clients.

Solution: don't. Cut the number of threads, and when a thread gets a connect timeout, don't create any more.

来源：https://stackoverflow.com/questions/42849629/httpurlconnection-is-not-giving-proper-response-code-in-case-of-multithreading

标签

java

multithreading

httpurlconnection