Logging HTML requests in robot framework

Question

I have been struggling to find much information to go along with this so I have turned here for help.

I am running UI tests of a web app using robot framework. When a test fails I want a log of the HTML requests so I can look back and see what failed, i.e. things not loading, 500 errors etc.

To this point I haven't managed to find something within the robot framework or selenium?

Another option is to see if there is a python library for logging this sort of thing or whether it would be a reasonable task to create one?

I have also looked into using autoit it use the browsers internal network logging tools but using these is a whole test of its own and I am not sure how well it would work. I am sure I must not be the first person to want this functionality?

I have continued to look into this and have found a viable option may be a packet sniffer using pcapy, I have no idea what to do in network programming and how I would proccess packets to only get post and get packets and repsonses, any help would be much appreciated

Cheers

Answer 1

Selenium is only emulating user behaviour, so it does not help you here. You could use a proxy that logs all the traffic and lets you examine the traffic. BrowserMob Proxy let's you do that. See Create Webdriver from Selenium2Libray on how to configure proxy for your browser.

This way you can ask your proxy to return the traffic after you noticed a failure in you test.

Answer 2

I have implemented same thing using BrowserMobProxy. It captures network traffic based on the test requirement.

First function CaptureNetworkTraffic(), will open the browser with configuration provided in the parameters.

Second function Parse_Request_Response(), will get the HAR file from above function and return resp. network data based on the parameter configured.

e.g.

print Capture_Request_Response("g:\\har.txt","google.com",True,True,False,False,False)

In this case, it will check url with "google.com" and returns response and request headers for the url.

    from browsermobproxy import Server
    from selenium import webdriver
    import json

    def CaptureNetworkTraffic(url,server_ip,headers,file_path):
    ''' 
    This function can be used to capture network traffic from the browser.                       Using this function we can capture header/cookies/http calls made from the      browser
url - Page url
    server_ip - remap host to for specific URL
    headers - this is a dictionary of the headers to be set
    file_path - File in which HAR gets stored
    '''
    port = {'port':9090}
    server = Server("G:\\browsermob\\bin\\browsermob-proxy",port) #Path to      the BrowserMobProxy
    server.start()
    proxy = server.create_proxy()
    proxy.remap_hosts("www.example.com",server_ip)
    proxy.remap_hosts("www.example1.com",server_ip)
    proxy.remap_hosts("www.example2.com",server_ip)
    proxy.headers(headers)
    profile  = webdriver.FirefoxProfile()
    profile.set_proxy(proxy.selenium_proxy())
    driver = webdriver.Firefox(firefox_profile=profile)
    new = {'captureHeaders':'True','captureContent':'True'}
    proxy.new_har("google",new)
    driver.get(url)
    proxy.har # returns a HAR JSON blob
    server.stop()
    driver.quit()
    file1 = open(file_path,'w')
    json.dump(proxy.har,file1)
    file1.close()


def Parse_Request_Response(filename,url,response=False,request_header=False,request_cookies=False,response_header=False,response_cookies=False):
resp ={}
har_data = open(filename, 'rb').read()
har = json.loads(har_data)
for i in har['log']['entries']:
    if url in i['request']['url']:
        resp['request'] = i['request']['url']
        if response:
            resp['response'] = i['response']['content']
        if request_header:
            resp['request_header'] = i['request']['headers']
        if request_cookies:
            resp['request_cookies'] = i['request']['cookies']
        if response_header:
            resp['response_header'] = i['response']['headers']
        if response_cookies:
            resp['response_cookies'] = i['response']['cookies']
return resp


  if (__name__=="__main__"):
  headers = {"User-Agent":"Mozilla/5.0 (iPad; CPU OS 5_0 like Mac OS X)  AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9A334 Safari/7534.48.3"}

    CaptureNetworkTraffic("http://www.google.com","192.168.1.1",headers,"g:\\har.txt")
    print Parse_Request_Response("g:\\har.txt","google.com",False,True,False,False,False)