You have to create your own IObjectMapper to map a custom collection like ISet
Create your own configuration instance with all the standard
objectmappers and your new
setobjectmapper.
Use an IMappingEngine instance created with the configuration with
your own objectmapper instead of the
static AutoMapper.Mapper class.
Some remarks:
It's easy to configure the IMappingEngine construction in a inversion of control container.
The source of automapper itself might help you with creating the IObjectMapper implementation.
You are using automapper on the opposite way for what it is designed for: It's designed to map complex objects to simple objects. You try to map a simple DTO to a complex entity. (This does not mean that what you want is hard to do with automapper, but you might get different problems in the future)
You are using the anemic domain model anti pattern. Domain should hold all the business logic, so it should not expose a complex collection like ISet (and no public setters for collections at all)