I've had success using PocketSphinx for this type of application. I'd suggest building the PocketSphinx Demo app, use the online sphinx lmtool to build language model of your short list of commands and see if it meets your response time needs. If the onResults callback isn't fast enough, you can use the onPartialResults callback which returns almost immediately--that's what I do and I'm happy with the performance. The demo app has a little built-in timer in the UI, too.
If you use the online lmtool, you can just take the lm file and the dict file from what it produces, using those to replace the corresponding lm and dict files in the language model they point you to in the demo setup instructions.
It was a bit of a pain to get built, but it was my first time using the NDK, which was finicky.
