Experienced also low performance with XDebug (loading Captcha in 6 seconds instead of milliseconds)
Remarks on this page got me on the way to identify the cause.
Turned off the profiler and loading time was divided by 3.
Still slow, but better already.