Removing(removing reference, not deleting) and then adding the appropriate file(the file of class you want to add the outlet to) is actually enough.
Edit 1 I found that after unchecking (in XCode 6.3.1) I had to wait a few seconds for the Indexing to appear and complete in the project name box. Same for the way backwards. This increased chances to fix the issue in almost all (but not all) cases.
Edit 2
Removing reference means that You do not delete the file completely but just remove it from the project (it still exist in the folder of your project, you add it later).
