Parse HTML source after login with Java

问题

I've been trying to access a website to parse data for an Android application I am developing, but I am having no luck when it comes to logging in.

The website is https://giffgaff.com/mobile/login

And below is a stripped out version of the form from that page (HTML):

<form action="/mobile/login" method="post">
    <input type="hidden" name="login_security_token" value="b22155c7259f402f8e005a771c460670">    
    <input type="hidden" name="redirect" value="/mobile">    
    <input type="hidden" name="p_next_page" value="">    


    <input name="nickname" maxlength="25" type="text" value="" />            
    <input name="password" type="password" value="" />

    <button name="step" type="submit" value="Login">Login</button>
</form>

Can anyone please suggest how I can login to this website using Java then parse the redirected page?

Up to now, I've tried processes on the lines of:

public static void main(Context context) {
    try {
        // Construct data
        String data = URLEncoder.encode("nickname", "UTF-8") + "=" + URLEncoder.encode("testingA", "UTF-8");
        data += "&" + URLEncoder.encode("password", "UTF-8") + "=" + URLEncoder.encode("testing", "UTF-8");

        // Send data
        URL url = new URL("https://giffgaff.com/mobile/login");
        URLConnection conn = url.openConnection();
        conn.setDoOutput(true);
        OutputStreamWriter wr = new OutputStreamWriter(conn.getOutputStream());
        wr.write(data);
        wr.flush();

        // Get the response
        BufferedReader rd = new BufferedReader(new InputStreamReader(conn.getInputStream()));
        String str = "";
        String line;
        while ((line = rd.readLine()) != null) {
            str += line;
        }

        AlertDialog alertDialog = new AlertDialog.Builder(context).create();
        alertDialog.setTitle("Output");
        alertDialog.setMessage(str);
        alertDialog.setButton("Okay", new DialogInterface.OnClickListener() {
            public void onClick(DialogInterface dialog, int which) {

            }
        });
        alertDialog.show();

        wr.close();
        rd.close();
    } catch (Exception e) {
        AlertDialog alertDialog = new AlertDialog.Builder(context).create();
        alertDialog.setTitle("ERROR");
        alertDialog.setMessage(e.toString());
        alertDialog.setButton("Okay", new DialogInterface.OnClickListener() {
            public void onClick(DialogInterface dialog, int which) {

            }
        });

        alertDialog.show();
    }
}

But my attempts return the page as if the login information was incorrect.

If you would like to see for yourself how the login page behaves, here's some test login details: Nickname (username): testingA Password: testing The site also seems to depend on a Cookie called "napaSessionId"

回答1:

First a word of caution, if you don't have direct permission to do this, beware, the site in question may preclude this in their terms of service.

To answer the question, there are many, many reasons a site would reject a login. To do this successfully you need to get as close as possible to how a browser would handle the transaction. To do that you need to see what a real browser is doing.

https is more tricky as many http sniffers can't deal with it but httpwatch claims it can. Check out the HTTP transactions and then try to replicate them.

Your url.openConnection() call will actually return an instance of HTTPURLConnction, cast to that & then you'll be able to easily set various http headers such as the User-Agent.

A final note, you say a cookie may be required. Your code isn't going to deal with cookies. To do that you'll need to use a cookie manager, e.g.: http://download.oracle.com/javase/tutorial/networking/cookies/index.html

回答2:

You might want to check out Jsoup, htmlUnit and httpUnit. I'm trying this right now and am confronted with all kinds of difficulties, but I'm sure one of those projects is the way to go...

Good luck, keep me posted!

来源：https://stackoverflow.com/questions/7741694/parse-html-source-after-login-with-java

标签

java

html-parsing