Presumably no one is forcing you to use a DI container framework. You're already using DI to decouple your classes and improve testability, so you're getting many of the benefits. In short, you're favoring simplicity, which is generally a good thing.
If your system reaches a level of complexity where manual DI becomes a chore (that is, increases maintenance), weigh that against the team learning curve of a DI container framework.
If you need more control over dependency lifetime management (that is, if you feel the need to implement the Singleton pattern), look at DI containers.
If you use a DI container, use only the features you need. Skip the XML configuration file and configure it in code if that is sufficient. Stick to constructor injection. The basics of Unity or StructureMap can be condensed down to a couple of pages.
There's a great blog post by Mark Seemann on this: When to use a DI Container
