Or simply use the inbuilt exclude option, which works regardless of whether the initial variable is a character or factor.
x <- c("a", "b", "c", "NotPerformed")
factor(x, exclude = "NotPerformed")
[1] a b c
Levels: a b c
factor(factor(x), exclude = "NotPerformed")
[1] a b c
Levels: a b c