I got the same error and resolved this by configuring it in the run.conf.bat
Run the JVM with the configuring run.conf.bat in Jboss5x
If free memory is not available AS you are passing in the statement then please make changes in run.conf.bat
set "JAVA_OPTS=-Xms512m -Xmx512m -XX:MaxPermSize=256m"