From reading the db:test tasks's source, it looks like they only care about grabbing the test db info from database.yml, but don't care which actual environment they're doing it under.
You might need to run rake db:test:prepare RAILS_ENV=test to ensure you're under the test environment.