How to write opencv mat to gstreamer pipeline?

Question

I want to add some opencv processes to a gstreamer pipeline and then send it over udpsink.

I\'m able to read frames from gstreamer like this:

// may          


        

Answer 1

Ok this is long for comment.. its not answer but few hints:

1a, Use netcast to check what is recieved on reciever side.. Its simple:

shell> nc -l 5000 -u

Than check whats being printed when you send something to the reciever.. nc is set to dump everything to the screen..

1b, you can try vlc for reciever and check the debug messages (its located in Tools > Messages - or hit Ctrl+M). Set the log lever to 2 debug .. Then open network stream and use udp://@:5000 as URL..

Btw you could test it with rtp with pipe:

appsrc ! x264enc ! mpegtsmux ! rtpmp2tpay ! udpsink host=localhost port=5000

Which is in vlc rtp://@:5000 then..

2, Check whats flowing after appsrc by using identity element (very helpful.. I use it often to debug pipe problems):

change your pipe to (note the identity element and -v for udpsink):

cv::VideoWriter writer = cv::VideoWriter("appsrc ! identity silent=false ! x264enc ! mpegtsmux ! udpsink -v host=localhost port=5000",  0, (double)30, cv::Size(640, 480), true);

Then run your code and check its output.. it shall list the incomming buffers from appsrc

3, To the code you posted as update - no I meant to use caps= attribute for caps, but maybe there is not difference:

    writer.open("appsrc caps="video/x-raw, framerate=30/1, width=640, height=480, format=RGB" ! autovideoconvert ! ximagesink sync=false", 0, (double)30, cv::Size(640, 480), true);

Answer 2

After hours of searching and testing, I finally got the answer. The key is to use only videoconvert after appsrc, no need to set caps. Therefore, a writer pipeline would look like appsrc ! videoconvert ! x264enc ! mpegtsmux ! udpsink host=localhost port=5000.

Following is a sample code that reads images from a gstreamer pipeline, doing some opencv image processing and write it back to the pipeline.

With this method, you can add any opencv process to a gstreamer pipeline easily.

// Compile with: $ g++ opencv_gst.cpp -o opencv_gst `pkg-config --cflags --libs opencv`

#include <stdio.h>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/opencv.hpp>

int main(int argc, char** argv) {

    // Original gstreamer pipeline: 
    //      == Sender ==
    //      gst-launch-1.0 v4l2src 
    //      ! video/x-raw, framerate=30/1, width=640, height=480, format=RGB 
    //      ! videoconvert
    //      ! x264enc noise-reduction=10000 tune=zerolatency byte-stream=true threads=4
    //      ! mpegtsmux 
    //      ! udpsink host=localhost port=5000
    //      
    //      == Receiver ==
    //      gst-launch-1.0 -ve udpsrc port=5000
    //      ! tsparse ! tsdemux 
    //      ! h264parse ! avdec_h264 
    //      ! videoconvert 
    //      ! ximagesink sync=false

    // first part of sender pipeline
    cv::VideoCapture cap("v4l2src ! video/x-raw, framerate=30/1, width=640, height=480, format=RGB ! videoconvert ! appsink");
    if (!cap.isOpened()) {
        printf("=ERR= can't create video capture\n");
        return -1;
    }

    // second part of sender pipeline
    cv::VideoWriter writer;
    writer.open("appsrc ! videoconvert ! x264enc noise-reduction=10000 tune=zerolatency byte-stream=true threads=4 ! mpegtsmux ! udpsink host=localhost port=9999"
                , 0, (double)30, cv::Size(640, 480), true);
    if (!writer.isOpened()) {
        printf("=ERR= can't create video writer\n");
        return -1;
    }

    cv::Mat frame;
    int key;

    while (true) {

        cap >> frame;
        if (frame.empty())
            break;

        /* Process the frame here */

        writer << frame;
        key = cv::waitKey( 30 );
    }
}

Hope this helps. ;)