Problem in HtmlUnit API for Java (Headless Browser)?

Question

I am using HtmlUnit headless browser to browse this webpage (you can see the webpage to have a better understanding of the problem).

I have set the select's value to "1"

by the following commands

final WebClient webClient = new WebClient(BrowserVersion.INTERNET_EXPLORER_7);
    try {
        // Configuring the webClient
        webClient.setJavaScriptEnabled(true);
        webClient.setThrowExceptionOnScriptError(false);
        webClient.setCssEnabled(true);
        webClient.setUseInsecureSSL(true);
        webClient.setRedirectEnabled(true);
        webClient.setActiveXNative(true);
        webClient.setAppletEnabled(true);
        webClient.setPrintContentOnFailingStatusCode(true);
        webClient.setAjaxController(new NicelyResynchronizingAjaxController());

        // Adding listeners
        webClient.addWebWindowListener(new com.gargoylesoftware.htmlunit.WebWindowListener() {

            public void webWindowOpened(WebWindowEvent event) {
                numberOfWebWindowOpened++;
                System.out.println("Number of opened WebWindow: " + numberOfWebWindowOpened);
            }

            public void webWindowContentChanged(WebWindowEvent event) {
            }

            public void webWindowClosed(WebWindowEvent event) {
                numberOfWebWindowClosed++;
                System.out.println("Number of closed WebWindow: " + numberOfWebWindowClosed);
            }
        });

        webClient.setWebConnection(new HttpWebConnection(webClient) {
            public WebResponse getResponse(WebRequestSettings settings) throws IOException {
                System.out.println(settings.getUrl());
                return super.getResponse(settings);
            }
        });

        CookieManager cm = new CookieManager();
        webClient.setCookieManager(cm);


        HtmlPage page = webClient.getPage("http://www.ticketmaster.com/event/0B004354D90759FD?artistid=1073053&majorcatid=10002&minorcatid=207");

        HtmlSelect select = (HtmlSelect) page.getElementById("quantity_select");
select.setSelectedAttribute("1", true);

and then clicked on the following button

by the following commands

HtmlButtonInput button = (HtmlButtonInput) page.getElementById("find_tickets_button");
HtmlPage captchaPage = button.click();
Thread.sleep(60*1000);
System.out.println("======captcha page=======");
System.out.println(captchaPage.asXml());

but even after clicking on the button and waiting for 60 seconds through the Thread.sleep() method, I am getting the same HtmlPage.

But when I do the same thing through real browser then I get the page that contains CAPTCHA.

I think I am missing something in the htmlunit.

Q1. Why am I not getting the same page (that contains CAPTCHA) through htmlunit's browser?

Answer 1

The web form on that page requires the quantity_select drop-down to be filled in. You're attempting to do that in your code by assuming the drop-down is a select element. However, it's no longer a select element. Try using Firebug to inspect the drop-down and you'll see that JavaScript has replaced the select with a complex set of nested div elements.

If you figure out how to emulate each user click on the divs for that unusual drop-down then you should be able to submit the form.