If Apple is still signing 5.0 firmware installations (which is unlikely but worth a try) then user1116644's instructions will work.
Otherwise, the only way to downgrade from 5.1 is if you have an SHSH blob from one of the iOS 4 releases saved using TinyUmbrella. Then you can use that to install using the same procedure as user1116644 described, but using the TinyUmbrella TSS server to do the signing (more info here: http://thefirmwareumbrella.blogspot.com/)
Unfortunately Apple changed the way signing works for iOS 5 and SHSH blobs can no longer be used to downgrade to 5.x firmwares, only 4.3.x and below.
Edit (4-30-12): Actually since I posted this, it looks like a workaround has been developed to allow iOS 5.x downgrading as long as you saved your SHSH blob.