Mock a series of interdependent calls

Question

I have a method that scrapes a web page and saves data into a file (see below for an example code). I need to test that the resulting data is well-formed.

The proble

Answer 1

For the code as it is, I agree, unit-testing the way you describe does not make much sense. But, that is not just because it would be a lot of work: The goal of testing is, certainly, to find errors in the code. The goal of unit-testing is, to find those errors that can be found in the isolated unit. But, a significant part of your example code is related to interaction with external libraries.

There is comparably little code on the algorithmic level, for example:

os.path.join(path, "%d.txt" % i)

or

u = u'http://example.com/%s/accreditation' % i

or the creation of the output file content.

That is, if there are bugs in the code, they are more likely to be on the interaction level: Calling the right library functions in the right order with the right parameters, parameters having the correct formats etc. - With mocks of the libraries, however, you will not find the interaction bugs, because the mocks are implemented by you and will just reflect your (potentially wrong) understanding of the library behaviour.

My suggestion for testing this code is: Separate the algorithmic code from the code that does the interaction with the libraries. You could, for example, create small helper functions to compute the output file name and the input url. You could, in the interaction dominated part of the code, extract all the data from the web page, and then (in a separate function) create the output file content using all that data.

These helper functions can then all be tested using unit-testing. The rest of the functionality you would test with integration testing.