If you will (or even might) destroy objects of a derived class through a base class pointer, you need a virtual destructor.
I take the approach that if I'm going to derive from a class AT ALL, then it shall have a virtual destructor. There are effectively no cases in the code I write where the performance implications of a virtual destructor matter, and even if it's not actually needed today, it might end up needing it in the future when the class is modified.
Basically: Put virtual on all base class destructors unless you have a good, well-thought out reason not to.
That's just another rule of thumb, but it's one that keeps you from making later mistakes.
