I am writing web app for java learning. Using which users may compile their code on my serwer + run that code. Compiling is easy with JavaCompiler:
Jav
Adding to Kaleb's answer, be sure to run the target JVM with a strict heap limit (eg. -Xmx16M). And, of course, you'll want to limit the number of JVMs running.