I want to remove all existing SDK versions as well as Xcode 4.
Xcode 4 Guide says to do this:
sudo /Xcode4/Library/uninstall-devtools --mode=all
If you have XCode 4.3.1, just Move To Trash the XCode.app file in the Applications folder.