You can also find out which package the binary comes from an download that packages source code.
On Debian (and Ubuntu and anything else that's based on Debian) you do that like this:
$ dpkg -S /usr/bin/factor
coreutils: /usr/bin/factor
$ apt-get source coreutils
The first command will check which package contains the file you are searching for (use "which factor" to find out which binary is executed when you just type "factor").
The second command will download and unpack the sources (including the patches applied to build the package) to the current directory, so it should be executed in a dedicated or temporary directory.
I'm pretty sure rpm-based distributions have a similar mechanism, but I don't know their commands.
