I have a sdl/opengl game I am working on for fun. I get a decent fps on average, but movement is really choppy because SDL_GL_SwapBuffers() will randomly take a crazy long a
SDL does use the SwapIntervalEXT extension so you can make sure that the buffer swaps are as fast as possible (VSYNC disabled). Also, buffer swap is not a simple operation, OpenGL needs to copy contents of back buffers to front buffers for case that you want to glReadPixels(). This behavior can be controlled using WGL_ARB_pixel_format, using WGL_SWAP_EXCHANGE_ARB (you can read about all this stuff in the specs; now I'm not sure if there is an alternative to that for Linux).
And then on top of all that, there is the windowing system. That can actually cause a lot of trouble. Also, if some errors are generated ...
This behavior is probably ok if you're running on a small mobile GPU.
SDL_GL_SwapBuffers() only contains a call to glxSwapBuffers() / wglSwapBuffers() so there is no time spent in there.