Installing third-party components always take a long time specially if you have large ones, but also it take more time if you setup the environment in more than one computer.
Firstly, I'll agree with both Ken and Fabricio that you must have the source code for all components you are using in a project. Anything else is just asking for trouble.
We don't use Subversion for our Source Control, but I'm guessing what we do would still be applicable...
Each project that we work on has a full copy of all components (source) used in that project. When we release, we create a release branch that includes the components as well as the project source. Each project includes it's own BPL directory.
We always create separate shortcuts to run Delphi for each project (or branch of a project) that we want to work on, and use the -R command-line parameter to set a unique Registry key for the Delphi settings for that project.
We then make sure we override the Path environment variable within Delphi to point to our project BPL directory instead of the normal Delphi BPL directory.
We set the BPL and DCP output directories for all components to be the local project BPL directory.
This allows us to have multiple versions of Delphi, with multiple versions of projects using different versions of components without any problems.