How can one mock/stub python module like urllib

Question

I need to test a function that needs to query a page on an external server using urllib.urlopen (it also uses urllib.urlencode). The server could be down, the page could cha

Answer 1

HTTPretty works in the exact same way that FakeWeb does. HTTPretty works in the socket layer, so it should work intercepting any python http client libraries. It's battle tested against urllib2, httplib2 and requests

import urllib2
from httpretty import HTTPretty, httprettified


@httprettified
def test_one():
    HTTPretty.register_uri(HTTPretty.GET, "http://yipit.com/",
                           body="Find the best daily deals")

    fd = urllib2.urlopen('http://yipit.com')
    got = fd.read()
    fd.close()

    assert got == "Find the best daily deals"

Answer 2

The simplest way is to change your function so that it doesn't necessarily use urllib.urlopen. Let's say this is your original function:

def my_grabber(arg1, arg2, arg3):
    # .. do some stuff ..
    url = make_url_somehow()
    data = urllib.urlopen(url)
    # .. do something with data ..
    return answer

Add an argument which is the function to use to open the URL. Then you can provide a mock function to do whatever you need:

def my_grabber(arg1, arg2, arg3, urlopen=urllib.urlopen):
    # .. do some stuff ..
    url = make_url_somehow()
    data = urlopen(url)
    # .. do something with data ..
    return answer

def test_my_grabber():
    my_grabber(arg1, arg2, arg3, urlopen=my_mock_open)

Answer 3

I am using Mock's patch decorator:

from mock import patch

[...]

@patch('urllib.urlopen')
def test_foo(self, urlopen_mock):
    urlopen_mock.return_value = MyUrlOpenMock()

Answer 4

Another simple approach is to have your test override urllib's urlopen() function. For example, if your module has

import urllib

def some_function_that_uses_urllib():
    ...
    urllib.urlopen()
    ...

You could define your test like this:

import mymodule

def dummy_urlopen(url):
    ...

mymodule.urllib.urlopen = dummy_urlopen

Then, when your tests invoke functions in mymodule, dummy_urlopen() will be called instead of the real urlopen(). Dynamic languages like Python make it super easy to stub out methods and classes for testing.

See my blog posts at http://softwarecorner.wordpress.com/ for more information about stubbing out dependencies for tests.

Answer 5

In case you don't want to even load the module:

import sys,types
class MockCallable():
  """ Mocks a function, can be enquired on how many calls it received """
  def __init__(self, result):
    self.result  = result
    self._calls  = []

  def __call__(self, *arguments):
    """Mock callable"""
    self._calls.append(arguments)
    return self.result

  def called(self):
    """docstring for called"""
    return self._calls

class StubModule(types.ModuleType, object):
  """ Uses a stub instead of loading libraries """

  def __init__(self, moduleName):
    self.__name__ = moduleName
    sys.modules[moduleName] = self

  def __repr__(self):
    name  = self.__name__
    mocks = ', '.join(set(dir(self)) - set(['__name__']))
    return "<StubModule: %(name)s; mocks: %(mocks)s>" % locals()

class StubObject(object):
  pass

And then:

>>> urllib = StubModule("urllib")
>>> import urllib # won't actually load urllib

>>> urls.urlopen = MockCallable(StubObject())

>>> example = urllib.urlopen('http://example.com')
>>> example.read = MockCallable('foo')

>>> print(example.read())
'foo'

Answer 6

Did you give Mox a look? It should do everything you need. Here is a simple interactive session illustrating the solution you need:

>>> import urllib
>>> # check that it works
>>> urllib.urlopen('http://www.google.com/')
<addinfourl at 3082723820L ...>
>>> # check what happens when it doesn't
>>> urllib.urlopen('http://hopefully.doesnotexist.com/')
#-- snip --
IOError: [Errno socket error] (-2, 'Name or service not known')

>>> # OK, let's mock it up
>>> import mox
>>> m = mox.Mox()
>>> m.StubOutWithMock(urllib, 'urlopen')
>>> # We can be verbose if we want to :)
>>> urllib.urlopen(mox.IgnoreArg()).AndRaise(
...   IOError('socket error', (-2, 'Name or service not known')))

>>> # Let's check if it works
>>> m.ReplayAll()
>>> urllib.urlopen('http://www.google.com/')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.5/site-packages/mox.py", line 568, in __call__
    raise expected_method._exception
IOError: [Errno socket error] (-2, 'Name or service not known')

>>> # yay! now unset everything
>>> m.UnsetStubs()
>>> m.VerifyAll()
>>> # and check that it still works
>>> urllib.urlopen('http://www.google.com/')
<addinfourl at 3076773548L ...>