setting request headers in selenium

Question

I\'m attempting to set the request header \'Referer\' to spoof a request coming from another site. We need the ability test that a specific referrer is used, which returns a

Answer 1

Webdriver doesn't contain an API to do it. See issue 141 from Selenium tracker for more info. The title of the issue says that it's about response headers but it was decided that Selenium won't contain API for request headers in scope of this issue. Several issues about adding API to set request headers have been marked as duplicates: first, second, third.

Here are a couple of possibilities that I can propose:

1. Use another driver/library instead of selenium
2. Write a browser-specific plugin (or find an existing one) that allows you to add header for request.
3. Use browsermob-proxy or some other proxy.

I'd go with option 3 in most of cases. It's not hard.

Note that Ghostdriver has an API for it but it's not supported by other drivers.

Answer 2

I wanted something a bit slimmer for RSpec/Ruby so that the custom code only had to live in one place. Here's my solution:

/spec/support/selenium.rb
...
RSpec.configure do |config|
  config.after(:suite) do
    $custom_headers = nil
  end
end

module RequestWithExtraHeaders
  def headers
    $custom_headers.each do |key, value|
      self.set_header "HTTP_#{key}", value
    end if $custom_headers

    super
  end
end
class ActionDispatch::Request
  prepend RequestWithExtraHeaders
end

Then in my specs:

/specs/features/something_spec.rb
...
$custom_headers = {"Referer" => referer_string}

Answer 3

check this out: chrome_options = Options()

chrome_options.add_argument('--headless')
chrome_options.add_argument('user-agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36"')

Problem solved!

Answer 4

For those people using Python, you may consider using Selenium Wire, which can set request headers, as well as provide you with the ability to inspect requests and responses.

from seleniumwire import webdriver  # Import from seleniumwire

# Create a new instance of the Firefox driver
driver = webdriver.Firefox()

# Set the request header using the `header_overrides` attribute
driver.header_overrides = {
    'Referer': 'referer_string',
}

# All subsequent requests will now contain the Referer
driver.get('https://mysite')

Answer 5

If you use the HtmlUnitDriver, you can set request headers by modifying the WebClient, like so:

final case class Header(name: String, value: String)

final class HtmlUnitDriverWithHeaders(headers: Seq[Header]) extends HtmlUnitDriver {
  super.modifyWebClient {
    val client = super.getWebClient
    headers.foreach(h => client.addRequestHeader(h.name, h.value))
    client
  }
}

The headers will then be on all requests made by the web browser.

Answer 6

I had the same issue. I solved it downloading modify-headers firefox add-on and activate it with selenium.

The code in python is the following

fp = webdriver.FirefoxProfile()
path_modify_header = 'C:/xxxxxxx/modify_headers-0.7.1.1-fx.xpi'
fp.add_extension(path_modify_header)

fp.set_preference("modifyheaders.headers.count", 1)
fp.set_preference("modifyheaders.headers.action0", "Add")
fp.set_preference("modifyheaders.headers.name0", "Name_of_header") # Set here the name of the header
fp.set_preference("modifyheaders.headers.value0", "value_of_header") # Set here the value of the header
fp.set_preference("modifyheaders.headers.enabled0", True)
fp.set_preference("modifyheaders.config.active", True)
fp.set_preference("modifyheaders.config.alwaysOn", True)

driver = webdriver.Firefox(firefox_profile=fp)