How to OpenGL on Linux Platform Without X-Window System, can I send OpenGL Graphics Directly to Framebuffer Device?
There Is Project Named DirectFB (Direct FrameBuff
As you said, you need a driver to do this. 99.9% of all Linux graphics drivers use X11, so unless you have a DirectFB OpenGL driver for your hardware, you are stuck with using X11.