What is the difference between the two and when one should be used instead of the other?
An Engine in rails terminology is a actually a subapplication of a web-application. For instance, something like a blog, a forum, or simple authentication: these are not full-blown applications, but pages/views/controllers/models that can be added to any rails application.
In rails2 this would be done using a plugin. Now since rails3 an engine can be packaged in a gem.
A gem is a ruby library, which can be found on http://rubygems.org and it is the standard (only) way to package and distribute ruby code to other rubyists.
So to conclude:
So when will you use one or the other:
Hope this helps.