My ZF2 application seems to be extremely slow when more than 3 users are using it at the same time.
I profile my code with xdebug and webgrind and non of my function
First of all to speedup your zf2 application you should use ZendOptimizerPlus. The vast part of execution time used to read and precompile php code. Typical ZF2 app has a lot of files, so it takes a lot of time to handle them.
ZendOp+ saves bytecode of your php application in shared memory, so server doesn't read a lot of files and doesn't parse it every request. ZendOp+ will be at php5.5 by default, so it is useful to know it and to use it.
Benchmarks gives 9x increase in performance for simple framework applications (symfony2 tests - http://www.ricardclau.com/2013/03/apc-vs-zend-optimizer-benchmarks-with-symfony2/ ).
I use it for my zf2 + doctrine2 + zfcUser application. Memcached is used for doctrine2 purposes, it gives only about 5% performance increase. So with ZendOp+ I got 6x increase (0.2 -> 0.03s) for simple pages and 3x increase (0.2 - 0.06) for complex pages with a lot of forms, entities, views. If I use classmap generator, I will update the answer.
Another issue is to use nginx + php-fpm rather than apache2+module. It saves server resources.