Get iplImage or Mat from directshow to opencv

Question

I had to change to directshow for my eyetracking software due to the difficulties to change resolution of the camera when using c++ and opencv.

Directshow is new to me and it is kind of hard to understand everything. But I found this nice example that works perfectly for capturing & viewing the web cam.

http://www.codeproject.com/Articles/12869/Real-time-video-image-processing-frame-grabber-usi

I am using the version that not requires directShow SDK. (But it is still directshow that is used in the example, right??)

#include <windows.h>
#include <dshow.h>

#pragma comment(lib,"Strmiids.lib")

#define DsHook(a,b,c) if (!c##_) { INT_PTR* p=b+*(INT_PTR**)a;   VirtualProtect(&c##_,4,PAGE_EXECUTE_READWRITE,&no);\
                                          *(INT_PTR*)&c##_=*p;   VirtualProtect(p,    4,PAGE_EXECUTE_READWRITE,&no);   *p=(INT_PTR)c; }


// Here you get image video data in buf / len. Process it before calling Receive_ because renderer dealocates it.
HRESULT ( __stdcall * Receive_ ) ( void* inst, IMediaSample *smp ) ; 
HRESULT   __stdcall   Receive    ( void* inst, IMediaSample *smp ) {     
    BYTE*     buf;    smp->GetPointer(&buf); DWORD len = smp->GetActualDataLength();
    HRESULT   ret  =  Receive_   ( inst, smp );   
    return    ret; 
}

int WINAPI WinMain(HINSTANCE inst,HINSTANCE prev,LPSTR cmd,int show){
    HRESULT hr = CoInitialize(0); MSG msg={0}; DWORD no;

    IGraphBuilder*  graph= 0;  hr = CoCreateInstance( CLSID_FilterGraph, 0, CLSCTX_INPROC,IID_IGraphBuilder, (void **)&graph );
    IMediaControl*  ctrl = 0;  hr = graph->QueryInterface( IID_IMediaControl, (void **)&ctrl );

    ICreateDevEnum* devs = 0;  hr = CoCreateInstance (CLSID_SystemDeviceEnum, 0, CLSCTX_INPROC, IID_ICreateDevEnum, (void **) &devs);
    IEnumMoniker*   cams = 0;  hr = devs?devs->CreateClassEnumerator (CLSID_VideoInputDeviceCategory, &cams, 0):0;  
    IMoniker*       mon  = 0;  hr = cams->Next (1,&mon,0);  // get first found capture device (webcam?)    
    IBaseFilter*    cam  = 0;  hr = mon->BindToObject(0,0,IID_IBaseFilter, (void**)&cam);
                               hr = graph->AddFilter(cam, L"Capture Source"); // add web cam to graph as source
    IEnumPins*      pins = 0;  hr = cam?cam->EnumPins(&pins):0;   // we need output pin to autogenerate rest of the graph
    IPin*           pin  = 0;  hr = pins?pins->Next(1,&pin, 0):0; // via graph->Render
                               hr = graph->Render(pin); // graph builder now builds whole filter chain including MJPG decompression on some webcams
    IEnumFilters*   fil  = 0;  hr = graph->EnumFilters(&fil); // from all newly added filters
    IBaseFilter*    rnd  = 0;  hr = fil->Next(1,&rnd,0); // we find last one (renderer)
                               hr = rnd->EnumPins(&pins);  // because data we are intersted in are pumped to renderers input pin 
                               hr = pins->Next(1,&pin, 0); // via Receive member of IMemInputPin interface
    IMemInputPin*   mem  = 0;  hr = pin->QueryInterface(IID_IMemInputPin,(void**)&mem);

    DsHook(mem,6,Receive); // so we redirect it to our own proc to grab image data

    hr = ctrl->Run();   

    while ( GetMessage(   &msg, 0, 0, 0 ) ) {  
        TranslateMessage( &msg );   
        DispatchMessage(  &msg ); 
    }
};

The method HRESULT Receive is called for every new frame from the cam. the the comments says that buf contains the data. But I have 3 problems/questions.

1. I cant include the opencv lib. I create a new project in visual studio, and add the same property sheets as I always include. the only difference from earlier projects is that I Now create a totaly empty project, earlier I created a win32 application. How to add opencv into the directshow project?

2. The example above. from buf. which is a pointer to the data. How do I get that into iplImage/Mat for the opencv calc?

3. Is there a way to not show the images from the webcam (I only need to perform some algorithms on the frames, I guess removing the window with the results might give me more power for the analyse algorithms?!)

Thanks!

Answer 1

With DirectShow you typically create a pipeline, that is a graph and you add filters to it, like this:

Camera -> [possibly some extra stuff] -> Sample Grabber -> Null Renderer

Camera, Sample Grabber, Null Renderer are all standard components shipped with clean Windows. Sample Grabber can be set to call you back via ISampleGrabberCB::SampleCB and give you data for every video frame captured. Null Renderer is the termination of pipeline without displaying video on monitor (just video capture).

SampleCB is the keyword to bring you sample code you need. Having data received with this call, you can convert/wrap it into IPL/OpenCV class as suggested by @praks411.

Having it done as simple as this, you don't need DirectShow BaseClasses, and the code will be merely regular ATL/MFC code and project. Make sure to use CComPtr wrapper class to deal with COM interfaces to not lose references and leak objects. Some declarations might be missing in very latest Windows SDK, so you need to either use Windows SDK 6.x or just copy missing parts from there.

See also:

• How to capture frames using Delphi/DSPack without displaying it on TVideoWindow? (Delphi code, but good description and figures)
• DirectShow: Examples for Using SampleGrabber for Grabbing a Frame and Building a VU Meter
• SetLifeCamStudioResolutionSample - A small DirectShow project showing how to set capture up, including resolution on camera, and Sample Grabber, and also missing SDK declarations; related Q is Can't make IAMStreamConfig.SetFormat() to work with LifeCam Studio
• Building the Filter Graph on Sample Grabber and Null Renderer

Answer 2

I think you can include opencv in existing. I've done that for console application. You will need to include path to opencv headers and path to opencv lib in property page for you current project.

Go to project property: 1.To addheaders C/C++ -----> Additional Include Directories ---> Here add opencv include directories (You may want to include multiples directories)

1. To add libs Linker -----> Additional Library Directories ----> Here add opencv lib.

To create IplImage from buf. You can use following once you have the height and width of image.

IplImage *m_img_show;
CvSize cv_img_size = cvSize(m_mediaInfo.m_width, m_mediaInfo.m_height);
            m_img_show = cvCreateImageHeader(cv_img_size, IPL_DEPTH_8U,3);
            cvSetData(m_img_show, m_pBuffer, m_mediaInfo.m_width*3);

I think preview of image is quite helpful. It seems that your filter above take data from renderer. If you do want you may want to change your renderer and use it in windowless mode. Other option could be to use sample grabber filter.