This is such a common scenario that there must be a sensible solution, yet despite pages of reading and copious Git gymnastics, my brain aches and I\'m unable to make this work.
Clearly you've been thinking about this, and I am only a novice at git, but my first instinct would be to add .gitignore to the plugins directory and have each plugin have it's own git repo. You could do the same for themes. So I guess this is more of a question than an answer - why wouldn't this simple approach work?