I\'m working at figuring out how to best work within my own repo for custom code while integrating with a vendor\'s library (in this case Magento). In my case, I will not n
Your question is more about git's submodule vs. subtree in general. I can't think of any Magento specifics that will influence the comparison. Most probably you are aware of subtree merging strategies which I will recommend but I am not sure why do you need to merge at a first place.
Best practice of merging is to avoid it and Magento architecture is flexible enough to allow it. Follow a simple rule set:
If your modification concerns PHP code:
If your modification concerns phtml templating -> use Magento layout mechanism to replace vendor's phtml with yours. A proper design customization will require heavy modification activities and layout work anyway.
If your modification concerns JS -> again, use layouts to link the code placed in js or skin folders.