JMX is definitely a viable solution. The EhCache doc has a page specifically for this.
Here's an example of configuring EhCache via JMX. The linked article contains a Spring config, but it's easily translatable to native Java if you're not using Spring.