What is the purpose of testing?
The majority of the answers so far are saying that private methods are implementation details which don't (or at least shouldn't) matter so long as the public interface is well-tested and working. That's absolutely correct if your only purpose for testing is to guarantee that the public interface works.
Personally, my primary use for code tests is to ensure that future code changes don't cause problems and to aid my debugging efforts if they do. I find that testing the private methods just as thoroughly as the public interface (if not more so!) furthers that purpose.
Consider: You have public method A which calls private method B. A and B both make use of method C. C is changed (perhaps by you, perhaps by a vendor), causing A to start failing its tests. Wouldn't it be useful to have tests for B also, even though it's private, so that you know whether the problem is in A's use of C, B's use of C, or both?
Testing private methods also adds value in cases where test coverage of the public interface is incomplete. While this is a situation we generally want to avoid, the efficiency unit testing depends both on the tests finding bugs and the associated development and maintenance costs of those tests. In some cases, the benefits of 100% test coverage may be judged insufficient to warrant the costs of those tests, producing gaps in the public interface's test coverage. In such cases, a well-targeted test of a private method can be a very effective addition to the code base.
