When I developed a piece of (academic) software using Java, I was forced to use an API that was rather badly implemented. This means that calls to this API for a certain set
I'm a big fan of separate processes for this kind of thing.
Spawn a sub process and wait for results.
If the API is non-deterministic, put the timer thread in a wrapper that makes the bad API into a main program.
That way, the subprocess always ends within the given time. It either produces a useful result or a system exit code that indicates failure.