My current project is to build a face authentication system. The constraint I have is: during enrollment, the user gives single image for training. However, I can add and us
To make your classifier robust you need to use condition independent features. For example, you cannot use face color since it depends on lighting conditions and state of a person itself. However, you can use distance between eyes since it is independent of any changes.
I would suggest building some model of such independent features and retrain classifier each time person starts authentication session. Best model I can think of is Active Appearance Model (one of implementations).