headless-browser

I have a very simple Puppeteer script that uses exposeFunction() to run something inside headless Chrome. (async function(){ var log = console.log.bind(console), puppeteer = require('puppeteer'); const browser = await puppeteer.launch(); const page = await browser.newPage(); var functionToInject = function(){ return window.navigator.appName; } await page.exposeFunction('functionToInject', functionToInject); var data = await page.evaluate(async function(){ console.log('woo I run inside a browser') return await functionToInject(); }); console.log(data); await browser.close(); })() This fails

I am running chrome the following way: google-chrome --headless --disable-gpu --dump-dom --disable-web-security Due to a corporate proxy, it seems to be bumping into a authentication popup that is asking to input credentials. How do I inject these credentials on the command line? They is no direct way to do this. Google don't allow connection to a proxy with login/password in headless mode. But, I was able to use proxy with credidential by using a MITMProxy. Other technology like Squid would work the same, but are more complicated to use. The trick is to create a local upstream proxy using

I have Debian Linux server that I use for a variety of things. I want it to be able to do some web-scraping jobs I need done regularly. This code can be found here . import sys from PyQt4.QtGui import * from PyQt4.QtCore import * from PyQt4.QtWebKit import * class Render(QWebPage): def __init__(self, url): self.app = QApplication(sys.argv, False) # Line updated based on mata's answer QWebPage.__init__(self) self.loadFinished.connect(self._loadFinished) self.mainFrame().load(QUrl(url)) self.app.exec_() def _loadFinished(self, result): self.frame = self.mainFrame() self.app.quit() A simple test

I have been researching about the headless browsers available till to date and found HtmlUnit being used pretty extensively. Do we have any alternative to HtmlUnit with possible advantage compared to HtmlUnit? Thanks Nayn As far as I know, HtmlUnit` is the most powerful headless browser. What are you issues with it? Sajid Hussain There are many other libraries that you can use for this. If you need to scrape xml base data use JTidy . If you need to scrape specific data from HTML you can use Jsoup . Well I use jsoup - it's pretty much faster than any other API. WebDriver with a virtual

I have used phantomjs but compatibility of phantom js is bad with protractor. Though in chrome and firefox browser I am able to find elements but same locators are not found in phantomjs. If you are using linux OS then you can use xvfb to launch the chrome browser in headless mode. To install xvfb, run the below commands in terminal sudo apt-get update sudo apt-get install -y xvfb Once you install xfvb, you can run your protractor scripts using the below command, xvfb-run protractor configuration.js The above command will start your chrome browser in headless mode and will execute all specs

How can i use Page.setDownloadBehavior in chrome dev tools to pass on so that I can set the download behavior for headless chrome with the code below? Create Chrome Browser [Arguments] ${link_to_open} ${chrome_options}= Evaluate sys.modules['selenium.webdriver'].ChromeOptions() sys, selenium.webdriver ${prefs}= Create Dictionary download.default_directory=${DOWNLOADS_DIR} Call Method ${chrome options} add_argument headless Call Method ${chrome options} add_argument disable-gpu Selenium2Library.Go To ${link_to_open} There appears to be a 'security feature' when trying to download a file using