I\'m using Swift 2 and using WeakContainer as a way to store a set of weak objects, much like NSHashTable.weakObjectsHashTable()
struct WeakCont         
        
I had the same idea to create weak container with generics.
As result I created wrapper for NSHashTable and did some workaround for your compiler error.    
class WeakSet: SequenceType {
    var count: Int {
        return weakStorage.count
    }
    private let weakStorage = NSHashTable.weakObjectsHashTable()
    func addObject(object: ObjectType) {
        guard object is AnyObject else { fatalError("Object (\(object)) should be subclass of AnyObject") }
        weakStorage.addObject(object as? AnyObject)
    }
    func removeObject(object: ObjectType) {
        guard object is AnyObject else { fatalError("Object (\(object)) should be subclass of AnyObject") }
        weakStorage.removeObject(object as? AnyObject)
    }
    func removeAllObjects() {
        weakStorage.removeAllObjects()
    }
    func containsObject(object: ObjectType) -> Bool {
        guard object is AnyObject else { fatalError("Object (\(object)) should be subclass of AnyObject") }
        return weakStorage.containsObject(object as? AnyObject)
    }
    func generate() -> AnyGenerator {
        let enumerator = weakStorage.objectEnumerator()
        return anyGenerator {
            return enumerator.nextObject() as! ObjectType?
        }
    }
}
  
Usage:
protocol MyDelegate : AnyObject {
    func doWork()
}
class MyClass: AnyObject, MyDelegate {
    fun doWork() {
        // Do delegated work.
    }
}
var delegates = WeakSet()
delegates.addObject(MyClass())
for delegate in delegates {
    delegate.doWork()
}
 
It's not the best solution, because WeakSet can be initialized with any type, and if this type doesn't conform to AnyObject protocol then app will crash. But I don't see any better solution right now.