You should use the "Add Library Package Reference" for all your external library on every project in your solution.
You'll end up with a packages.config per project.
However, you'll download the package only one time and reuse them locally for all your other projects.