My master branch layout is like this:
/ <-- top level
/client <-- desktop client source files
/server
You can have two git repositories (client and server) and add them to a "super-project" (app). In this "super-project" you can add the two repositories as submodules (check this tutorial).
Another possible solution (a bit more dirty) is to have separate branches for client and server, and then you can pull from the 'server' branch.