It seems to be impossible to make a cached thread pool with a limit to the number of threads that it can create.
Here is how static Executors.newCachedThreadPool is
Unless I've missed something, the solution to the original question is simple. The following code implements the desired behavior as described by the original poster. It will spawn up to 5 threads to work on an unbounded queue and idle threads will terminate after 60 seconds.
tp = new ThreadPoolExecutor(5, 5, 60, TimeUnit.SECONDS,
new LinkedBlockingQueue());
tp.allowCoreThreadTimeOut(true);