This really depends on the granularity of the actual differences between the implementations for the particular platforms (in your code this corresponds to "common stuff below here").
When the differences are considerable, I often prefer to factor out all platform-dependent code to several components (.cpp) sharing the same platform independent interface (.hpp).
For example, suppose we need to acquire images from a firewire camera within a cross-platform application program. Ways to achieve that on different platforms are hopelessly different. Then it makes sense to have a common interface which would be placed in CameraIeee1394.hpp, and to place the two platform dependent implementations into CameraIeee1394_unix.cpp and CameraIeee1394_w32.cpp.
The script which prepares the makefile for a particular platform can automatically disregard source files for foreign platforms by simply checking the filename suffix.
