I found daemonsy's reply to be very helpful. Here are a few other things to consider as you do this.
Replacing Devise
If you are going to replace Devise with your own authentication, I recommend this Railscast: Authentication from Scratch (revised) (subscription required, but it's the best $9/mo you can spend).
And this Railscast (no subscription required) can help with a forgot password link and "remember me" option (things Devise offers out of the box, but that you can build pretty easily yourself): Remember Me & Reset Password
Tests
Before you do this, I recommend running all your tests to make sure they're passing.
After you remove Devise, your authentication-dependent tests will probably fail, so plan to spend some time fixing failing tests. This is a good thing because it will help you see what stuff "broke" when you removed Devise.
Make sure you check your test helpers as well. Most of my helpers were in /spec/spec_helper.rb. In fact, most of my failing tests began passing once I updated the methods in spec_helper.rb (eg, "login_user").