A rails application in production should use smtp or sendmail to send mails?
As NDP already mentioned, they both work fine - that is, if your volume of messages doesn't exceed a certain amount.
For example, if your application can talk SMTP to either the local SMTP server (on IP 127.0.0.1, Port 25) or a server in the same subnet (i.e., over a low-latency link), and that server does not use any content filters before it queues a message, you will usually be able to submit a lot more mails over SMTP in a shorter time.
A useful link for Postfix may be General Mail Delivery Performance Tips - note the quote saying
Submit mail via SMTP instead of /usr/sbin/sendmail.
However, on modern hardware, if you don't plan to submit more than about 10 messages per second, you shouldn't notice any real difference.