After reading this answer, it looks like it is a best practice to use smart pointers as much as possible, and to reduce the usage of \"normal\"/raw pointers to minimum.
One instance where reference counting (used by shared_ptr in particular) will break down is when you create a cycle out of the pointers (e.g. A points to B, B points to A, or A->B->C->A, or etc). In that case, none of the objects will ever be automatically freed, because they are all keeping each other's reference counts greater than zero.
For that reason, whenever I am creating objects that have a parent-child relationship (e.g. a tree of objects), I will use shared_ptrs in the parent objects to hold their child objects, but if the child objects need a pointer back to their parent, I will use a plain C/C++ pointer for that.