403 error while fetching content from URL

Question

I am trying to automate a process. For that I need to fetch XML by hitting a URL, multiple times in 1 run, and then parse it. For 1 run of the program, the URL could be hit anywhere between 4 to 25 times. This all seems fine until a 403 error response is returned.

Interestingly, the 403 always comes up for every 5th or 6th time the URL is hit.

I am using JDOM to parse the XML response.

I have tried the codes:

Document doc = builder.build(new InputSource(url.openStream()));

and

HttpURLConnection conn = (HttpURLConnection)url.openConnection();
conn.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB;     rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13 (.NET CLR 3.5.30729)"); 
Document doc = builder.build(conn.getInputStream());

With the second one I get the Exception:

org.jdom.input.JDOMParseException: Error on line 1: White spaces are required between publicId and systemId.

Could someone please help me in getting rid of the 403. Please note that I do not have any control over the source if a change is required to be made as talked about here

Also, I am not sure if this link is helpful.

Thank you.

---

[UPDATE 1]: This is somehow working, without having to sleep:

try{
            doc = builder.build(conn.getInputStream());
        }catch(IOException ioEx){
            doc = builder.build(new InputSource(url.openStream()));
}

Answer 1

403 means that the request is understood but the server refuses to process it. Look the headers you send. And when fails run a TRACE http method to retrieve the exact petition you are performing.

When you stablish an http connection you send along with the request the method you want to perform.

One of this methods is TRACE.

By performing a TRACE method you can see in the body response the petition you just performed. So you can see if it is still valid.

Maybe you are exceeding the max number of petitions if they had any mechanism.