Is it possible?
If you download the source code, it will generally have a Makefile.PL. You run "perl Makefile.PL; make; make test; make install" and it will build and install for you.
Obviously if you're not using CPAN.pm, you're going to have to deal with dependencies yourself.
Also, if the reason you can't use CPAN.pm is that you don't have permission to install into /usr/lib/perl, you can force CPAN.pm to install locally, but I forget how.