I am having this annoying problem which takes 10-20 seconds and sometimes more every time my android project is refreshed or rebuilt.
Instead of attaching a source folder to the .jar file, you should zip the source folder and attach the zipped file to the .jar.
See this answer for more info https://stackoverflow.com/a/9699242/637754