So I have function that formats a date to coerce to given enum DateType{CURRENT, START, END} what would be the best way to handling return value with cases that use switch stat
I think that throw new IllegalArgumentException("Something strange happend") is the best pratice.
Using null will just presumibly cause a NullPointerException somewhere when you use the return value but it will be less informative than raising a specific exception that describes the problem!
And you know: clear errors = better developing.
I would go with the first approach (but with IllegalArgumentException as in your second approach). You should include a default statement to guard against cases when someone modifys (extends) your enum. Putting the exception in the default-statement makes clear to the reader that the code is never supposed to get past the switch-statement. Otherwise they would have to check if really all of the enum values are in the switch.
Exceptions, as you can obey more to the parent than a single return int can. Usually you use Exceptions where they exist (C++), and return values where not (C).
Throw an exception, since this is an exceptional case.
And throw it outside the switch, it would be more readable. Otherwise it sounds like "the default case is exceptional".