Is it bad practice to run tests on a database instead of on fake repositories?

Question

I know what the advantages are and I use fake data when I am working with more complex systems.

What if I am developing something simple and I can easily set up my e

Answer 1

It rather depends on whether the DB is automatically set up by the test, also whether the database is isolated from other developers.

At the moment it may not be a problem (e.g. only one developer). However (for manual database setup) setting up the database is an extra impediment for running tests, and this is a very bad thing.

Answer 2

Assuming that you want to automate this, the most important thing is that you can programmatically generate your initial condition. It sounds like that's the case, and even better you're testing real world data.

However, there are a few drawbacks:

Your real database might not cover certain conditions in your code. If you have fake data, you cause that behavior to happen.

And as you point out, you have a simple application; when it becomes less simple, you'll want to have tests that you can categorize as unit tests and system tests. The unit tests should target a simple piece of functionality, which will be much easier to do with fake data.

Answer 3

The reasons to use fake data instead of a real DB are:

1. Speed. If your tests are slow you aren't going to run them. Mocking the DB can make your tests run much faster than they otherwise might.
2. Control. Your tests need to be the sole source of your test data. When you use fake data, your tests choose which fakes you will be using. So there is no chance that your tests are spoiled because someone left the DB in an unfamiliar state.
3. Order Independence. We want our tests to be runnable in any order at all. The input of one test should not depend on the output of another. When your tests control the test data, the tests can be independent of each other.
4. Environment Independence. Your tests should be runnable in any environment. You should be able to run them while on the train, or in a plane, or at home, or at work. They should not depend on external services. When you use fake data, you don't need an external DB.

Now, if you are building a small little application, and by using a real DB (like MySQL) you can achieve the above goals, then by all means use the DB. I do. But make no mistake, as your application grows you will eventually be faced with the need to mock out the DB. That's OK, do it when you need to. YAGNI. Just make sure you DO do it WHEN you need to. If you let it go, you'll pay.

Answer 4

The downsides to running tests against your database is lack of speed and the complexity for setting up your database state before running tests.

If you have control over this there is no problem in running the tests directly against the database; it's actually a good approach because it simulates your final product better than running against fake data. The key is to have a pragmatic approach and see best practice as guidelines and not rules.

Answer 5

If you're just writing a simple one-off application that you absolutely know will not grow, I think a lot of "best practices" just go right out the window.

You don't need to use DI/IOC or have unit tests or mock out your db access if all you're writing is a simple "Contact Us" form. However, where to draw the line between a "simple" app and a "complex" one is difficult.

In other words, use your best judgment as there is no hard-and-set answer to this.

Answer 6

I think it depends on whether your queries are fixed inside the repository (the better option, IMO), or whether the repository exposes composable queries; for example - if you have a repository method:

IQueryable<Customer> GetCustomers() {...}

Then your UI could request:

var foo = GetCustomers().Where(x=>SomeUnmappedFunction(x));

bool SomeUnmappedFunction(Customer customer) {
   return customer.RegionId == 12345 && customer.Name.StartsWith("foo");
}

This will pass for an object-based fake repo, but will fail for actual db implementations. Of course, you can nullify this by having the repository handle all queries internally (no external composition); for example:

Customer[] GetCustomers(int? regionId, string nameStartsWith, ...) {...}

Because this can't be composed, you can check the DB and the UI independently. With composable queries, you are forced to use integration tests throughout if you want it to be useful.