A rails application in production should use smtp or sendmail to send mails?
They both will work fine. Action Mailer supports both.
I have used SMTP on several projects successfully. My sense was that this was a little more "standard", but I may be wrong.
I haven't used sendmail. My concern would be that it may be harder to set up in a development environment if you aren't developing directly on Unix/Linux. Where you can talk directly to any SMTP server -- even a remote one, -- you would have to install sendmail on the rails machine to get it to work.
The main problem I run into with email is sending messages asynchronously. Without a local SMTP server, a local sendmail instance is going to be more performant.
Either way, it looks pretty easy to switch if you decide you picked wrong.