htmlunit | 易学教程

Scraping flash using HtmlUnit or other java tool

阅读更多关于 Scraping flash using HtmlUnit or other java tool

问题 I am using HtmlUnit to scrap date from a site but after login all the data is displayed using adobe flash player as swf object, I don't know any way to scrap data from such page. Is there any way to extract data from flash page, If yes please help me out, either using HtmlUnit or any other java tool. thanks. 回答1: There is no way that you can interact with Flash applet using HtmlUnit. You can try with selenium; I never used it, but it looks like there are plugins that enable flash

How to limit HttpClient response length

阅读更多关于 How to limit HttpClient response length

问题 I'm using htmlunit with httpclient . How can I limit the response body length of httpclient to say 1MB ? 回答1: The trick is easy. You have to take the InputStream,read from it and stop reading when the limit is exceeded. InputStream instream = method.getResponseBodyAsStream(); I have done an example tuning the apache example a bit. import java.io.ByteArrayOutputStream; import java.io.IOException; import java.io.InputStream; import org.apache.commons.httpclient.DefaultHttpMethodRetryHandler;

HtmlUnitDriver not getting page properly

阅读更多关于 HtmlUnitDriver not getting page properly

问题 I'm a newbie at this, basically I'm trying to use the HtmlUnitDriver, this is my code: WebDriver driver = new HtmlUnitDriver(); driver.get("http://www.google.com"); System.out.println(driver.getPageSource()); But the page source I got is: <?xml version="1.0" encoding="UTF-8"?> <html> <head/> <body/> </html> I have tried to to new HtmlUnitDriver(true) but it's still not loading google I have already add the selenium server stand alone to the class path. Am I doing anything wrong? Thank you P.S

How to debug Htmlunit traffic with Fiddler using PAC (proxy auto-config)

阅读更多关于 How to debug Htmlunit traffic with Fiddler using PAC (proxy auto-config)

问题 I have an application using Htmlunit and need put Fiddler to intercept traffic, i read something about configure it via PAC (proxy auto-config) javascript file that comes with but i cant found the article again. How to configure Htmlunit via PAC ? Where the PAC javascript is located ? Thanks 回答1: "Fiddler is an HTTP Proxy running on port 8888 on your local PC. You can configure any application which accepts a HTTP Proxy to run through Fiddler so you can debug its traffic." (hookup) Try it:

Javascript based dynamic content using htmlUnit

阅读更多关于 Javascript based dynamic content using htmlUnit

问题 I have been stuck in getting JavaScript based dynamic content using HtmlUnit. I am expecting to get (Signin, Registration html content) from the page. With the following code, I only get the static content. I am new to HtmlUnit. Any help will be highly appreciated. String strURL = "https://www.checkmytrip.com" ; java.util.logging.Logger.getLogger("com.gargoylesoftware.htmlunit").setLevel(java.util.logging.Level.OFF); java.util.logging.Logger.getLogger("org.apache.http").setLevel(java.util

Login check using HtmlUnit

阅读更多关于 Login check using HtmlUnit

问题 Hy... i want to login to some 3rd party sites using HtmlUnit. But HtmlUnit should be able to tell me whether the login attempt to the input site is successful or not. Is there any way around to perform this task using HtmlUnit. Please help ..!!! Thanks Usman Raza 回答1: I'm currently using HTMLunit to log in to a site that has a varification page and redirect. some of my code for this is: //---------------------------------Login Page--------------------------------- HtmlPage PageLogin =

HtmlUnitDriver causes problems while getting an url

阅读更多关于 HtmlUnitDriver causes problems while getting an url

问题 I have a page crawler developed in Java using Selenium libraries. The crawler goes through a website that launches through Javascript 3 applications which are displayed as HTML in popup windows. The crawler has no issues when launching 2 of the applications, but on the 3rd one the crawler freezes forever. The code I'm using is similar to public void applicationSelect() { ... //obtain url by parsing tag href attributed ... this.driver = new HtmlUnitDriver(BrowserVersion.INTERNET_EXPLORER_8);

Using Webdriver for PrimeFaces file upload

阅读更多关于 Using Webdriver for PrimeFaces file upload

问题 I've got a problem writing a test using Webdriver and HTMLUnit for my Primefaces page. What I've done is to add a simple Primefaces fileupload to the page, which will take a CSV file (no validation as yet), like this: <p:fileUpload id="listFileUpload" mode="simple" value="#{fileImportView.file}" /> This will indeed make an UploadedFile object available to my listener method when used from Firefox. However, when the same listener is called through the test the resulting UploadedFile is null.

HtmlUnit and XPath: DOMNode.getByXPath only works on HtmlPage?

阅读更多关于 HtmlUnit and XPath: DOMNode.getByXPath only works on HtmlPage?

I'm trying to parse a page with links to articles whose important content looks like this: <div class="article"> <h1 style="float: none;"><a href="performing-arts">Performing Arts</a></h1> <a href="/performing-arts/EIF-theatre-review-Sin-Sangre.6517348.jp"> <span class="mth3"> <span id="wctlMiniTemplate1_ctl00_ctl00_ctl01_WctlPremiumContentIcon1"> </span> EIF theatre review: Sin Sangre | The Man Who Fed Butterflies | Caledonia | Songs Of Ascension | Vieux Carré | The Gospel At Colonus </span> <span class="mtp">The EIF's theatre programme wasn't as far-reaching as it could have been, but did

Java and HTMLUnit: How to click on submit button?

阅读更多关于 Java and HTMLUnit: How to click on submit button?

问题 I am brand new to Java and need to write various java applications to do web scraping and web page interaction. I started using Selenium but because it interacts directly with a browser, it is not practical for my use. I need to do the following tasks: 1. Go to a specific URL 2. Enter a post code in a input field 3. Click submit button 4. Parse and save results from specific div tag or re-query page. I am using HTMLUnit and Eclipse. I can access a webpage and enter a post code in an input by