I think it's necessary to perform the following optimisations:
Server-side:
- Use fast web-server like nginx or lighttpd for serving js and css files if not using yet.
- Enable caching and gziping for the files.
And client-side:
- Place all common js-files into minified one and enable hardcore caching. If your page don't require running them before "onload", you can add one to page dynamically. It will speed up loading.
- Place per-page code into separate files and if it's not necessary to do smth. before onload event, add it dynamically.
- If it more than a few Kbytes of css at all, just merge it into one. Else you should split it to common and per-page styles.
- If you need best performance on client-side, place into common css file really common styles, that applied to ALL pages. Load rules you need on the page only, it will speed up rendering.
- The simplest way to minify js and css together is to use YUI Compress. If you want better speedup, you can remove all evals and other "dynamic" code and use Google Closure Compiler.
If it will not help, than everything's bad and you need more servers to serve js and css files.