The shape of a program should reflect
only the problem it needs to solve.
And what happens when requirements change and your module wasn't abstracted using a Facade or potentially a Mediator, thus making it overly difficult to swap out?
design patterns are a sign of not
enough abstraction
Chances are, if you've abstracted everything properly, then you have a design pattern in there somewhere.
What if there is an extremely 'heavy' object that does not have to be loaded? The Proxy pattern saves the user from waiting forever.
I could go on, but I think I have said enough. Design patterns are great tools when used correctly. The problem comes when they are used improperly, but I guess that's why misused patterns are called anti-patterns.