Rendering a OpenCV IplImage to a QGLWidget

Question

I am having trouble with rendering a OpenGL scene.

The background is I want to display the frames from a video capture device in a preview window. I am using OpenCV and Qt. And to test I am capturing from my MacBook webcam. The preview window is 200x200 and the frame captured is 640x480. I am not worried about maintaining aspect ratios.

Other info from the IplImage struct:

• Debug: channels: 3
• Debug: depth: 8
• Debug: dataOrder: 0
• Debug: align: 4. Alignment of image rows 4 or 8
• Debug: origin: 0. 0=top left, 1=bottom left
• Debug: widthStep: 2560.
• Debug: colorModel: RGB

So this image shows the current state of affairs.

Current capture http://clinsoftsolutions.com/fgvc4.png

I started with using glDrawPixels, but this didn't work well. I got output, but no scaling.

Currently I am trying with textures and here is the code I am using for the GL interactions

void VideoCaptureWidget::initializeGL()
{
    qDebug("initializeGL called");
    qglClearColor(QColor::fromRgb(0,0,0));      // set clear colur to black
    glDisable(GL_DEPTH_TEST);
    glMatrixMode(GL_PROJECTION);
    glLoadIdentity();
    glOrtho(0, this->width(), this->height(), 0.0f, 0.0f, 1.0f);
    glMatrixMode(GL_MODELVIEW);
    glLoadIdentity();

    glEnable(GL_TEXTURE_2D);
    glGenTextures(3, &m_texture);
    glBindTexture(GL_TEXTURE_2D, m_texture);
    glTexParameteri(GL_TEXTURE_2D,GL_TEXTURE_MAG_FILTER,GL_NEAREST);
    glTexParameteri(GL_TEXTURE_2D,GL_TEXTURE_MIN_FILTER,GL_NEAREST);
    glBindTexture(GL_TEXTURE_2D,m_texture);
    glTexImage2D(GL_TEXTURE_2D,0,GL_RGB,this->width(),this->height(),0,GL_BGR,GL_UNSIGNED_BYTE,NULL);
    glDisable(GL_TEXTURE_2D);
}


void VideoCaptureWidget::paintGL()
{    
    glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
    glDisable(GL_DEPTH_TEST);

    glMatrixMode(GL_PROJECTION);
        glLoadIdentity();
        glOrtho(0.0f,this->width(),this->height(),0.0f,0.0f,1.0f);
        glMatrixMode(GL_MODELVIEW);
        glLoadIdentity();
        glEnable(GL_TEXTURE_2D);
    glBindTexture(GL_TEXTURE_2D,m_texture);
    glTexImage2D(GL_TEXTURE_2D,0,GL_RGB,m_image->width,m_image->height,0,GL_BGR_EXT,GL_UNSIGNED_BYTE,m_image->imageData);
    glBegin(GL_QUADS);
    glTexCoord2i(0,1); glVertex2i(0,this->height());
    glTexCoord2i(0,0); glVertex2i(0,0);
    glTexCoord2i(1,0); glVertex2i(this->width(),0);
    glTexCoord2i(1,1); glVertex2i(this->width(),this->height());
    glEnd();
    glFlush();

}

void VideoCaptureWidget::resizeGL(int width,int height)
{
    qDebug("reszieGL called");

    glViewport(0,0,this->width(),this->height());
    glMatrixMode(GL_PROJECTION);
    glLoadIdentity();
    glOrtho(0.0f,this->width(),this->height(),0.0f,0.0f,1.0f);
    glMatrixMode(GL_MODELVIEW);
    glLoadIdentity();
 }

The image is captured in a timer slot and m_image is a _IplImage pointer

void VideoCaptureWidget::_captureFrame() {
    m_image = cvQueryFrame(m_capture);
    if(!m_image) {
        qDebug("VideoCaptureWidget::_captureFrame(): Error capturing a frame...");
    }

    //Draw the scene
    glDraw();
}

I am really hoping someone will have seen this sort of distorted image before and know what the problem is.

Answer 1

Another approach is to convert the BGR frames to RGBA before uploading them to the GPU with glTexImage2D(). I uploaded a complete demo in my repository, check cvQTcameraGL.

Here's the relevant code:

// Note: trying to retrieve more frames than the camera can give you
// will make the output video blink a lot.
cv_capture >> cv_frame;
if (cv_frame.empty())
{
    std::cout << "GLWidget::paintGL: !!! Failed to retrieve frame" << std::endl;
    return;
}
cv::cvtColor(cv_frame, cv_frame, CV_BGR2RGBA);

glEnable(GL_TEXTURE_RECTANGLE_ARB);

// Typical texture generation using data from the bitmap
glBindTexture(GL_TEXTURE_RECTANGLE_ARB, _texture);

// Transfer image data to the GPU
glTexImage2D(GL_TEXTURE_RECTANGLE_ARB, 0,
             GL_RGBA, cv_frame.cols, cv_frame.rows, 0,
             GL_RGBA, GL_UNSIGNED_BYTE, cv_frame.data);
if (glGetError() != GL_NO_ERROR)
{
    std::cout << "GLWidget::paintGL: !!! Failed glTexImage2D" << std::endl;
}

Answer 2

A bit more thinking and looking at the image I figured it must be a byte / channel alignment problem. So a bit more googling pointed mt in the direction of

glPixelStorei with GL_UNPACK_ALIGNMENT and then also GL_UNPACK_ROW_LENGTH

The final code that now works is:

glPixelStorei(GL_UNPACK_ALIGNMENT, 4 );
glPixelStorei(GL_UNPACK_ROW_LENGTH, m_image->widthStep/m_image->nChannels);

This needs to be added above the glTexImage2D call.

It's a weird format for the image data. But that is the output you get from the Apple iSight built in webcam. Hope this helps someone else.