How to write opencv mat to gstreamer pipeline?

Question

I want to add some opencv processes to a gstreamer pipeline and then send it over udpsink.

I'm able to read frames from gstreamer like this:

// may add some plugins to the pipeline later
cv::VideoCapture cap("v4l2src ! video/x-raw, framerate=30/1, width=640, height=480, format=RGB ! videoconvert ! appsink");
cv::Mat frame;
while(ture){
  cap >> frame;
  // do some processing to the frame
}

But what can't figure out is how to pass the processed frame to the following pipeline: appsrc ! x264enc ! mpegtsmux ! udpsink host=localhost port=5000

I've tried

cv::VideoWriter writer = cv::VideoWriter("appsrc ! x264enc ! mpegtsmux ! udpsink host=localhost port=5000",  0, (double)30, cv::Size(640, 480), true);
writer << processedFrame;

However, the receiver side receives nothing. (I uses the pipeline $gst-launch-1.0 udpsrc port=5000 ! tsparse ! tsdemux ! h264parse ! avdec_h264 ! videoconvert ! ximagesink sync=false as receiver)

My question is, can I pass processed opencv Mat to a gstreamer pipeline and let it do some encoding, and then send over network through udpsink? If yes, how do I achieve this?

Side question, is there any way I can debug a VideoWriter? Such as checking if frames are actually written into it.

Note that I'm using opencv 2.4.12 and gstreamer 1.2 on ubuntu 14.04.

Any help are great!

EDIT: To provide more info, I tested the following code, and it gave GStreamer Plugin: Embedded video playback halted; module appsrc0 reported: Internal data flow error.

#include <stdio.h>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/opencv.hpp>

int main(int argc, char *argv[]){
    cv::VideoCapture cap("v4l2src ! video/x-raw, framerate=30/1, width=640, height=480, format=RGB ! videoconvert ! appsink");
    if (!cap.isOpened()) {
        printf("=ERR= can't create capture\n");
        return -1;
    }
    cv::VideoWriter writer;
    // problem here
    writer.open("appsrc ! video/x-raw, framerate=30/1, width=640, height=480, format=RGB ! autovideoconvert ! ximagesink sync=false", 0, (double)30, cv::Size(640, 480), true);
    if (!writer.isOpened()) {
        printf("=ERR= can't create writer\n");
        return -1;
    }

    cv::Mat frame;
    int key;

    while (true) {
        cap >> frame;
        if (frame.empty()) {
            printf("no frame\n");
            break;
        }
        writer << frame;
        key = cv::waitKey( 30 );
    }

    cv::destroyWindow( "video" );
}

Apparently there's something wrong with the appsrc pipeline, but I have no idea what went wrong because the pipeline gst-launch-1.0 v4l2src ! video/x-raw, framerate=30/1, width=640, height=480, format=RGB ! videoconvert ! ximagesink sync=false works fine.

Answer 1

After hours of searching and testing, I finally got the answer. The key is to use only videoconvert after appsrc, no need to set caps. Therefore, a writer pipeline would look like appsrc ! videoconvert ! x264enc ! mpegtsmux ! udpsink host=localhost port=5000.

Following is a sample code that reads images from a gstreamer pipeline, doing some opencv image processing and write it back to the pipeline.

With this method, you can add any opencv process to a gstreamer pipeline easily.

// Compile with: $ g++ opencv_gst.cpp -o opencv_gst `pkg-config --cflags --libs opencv`

#include <stdio.h>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/opencv.hpp>

int main(int argc, char** argv) {

    // Original gstreamer pipeline: 
    //      == Sender ==
    //      gst-launch-1.0 v4l2src 
    //      ! video/x-raw, framerate=30/1, width=640, height=480, format=RGB 
    //      ! videoconvert
    //      ! x264enc noise-reduction=10000 tune=zerolatency byte-stream=true threads=4
    //      ! mpegtsmux 
    //      ! udpsink host=localhost port=5000
    //      
    //      == Receiver ==
    //      gst-launch-1.0 -ve udpsrc port=5000
    //      ! tsparse ! tsdemux 
    //      ! h264parse ! avdec_h264 
    //      ! videoconvert 
    //      ! ximagesink sync=false

    // first part of sender pipeline
    cv::VideoCapture cap("v4l2src ! video/x-raw, framerate=30/1, width=640, height=480, format=RGB ! videoconvert ! appsink");
    if (!cap.isOpened()) {
        printf("=ERR= can't create video capture\n");
        return -1;
    }

    // second part of sender pipeline
    cv::VideoWriter writer;
    writer.open("appsrc ! videoconvert ! x264enc noise-reduction=10000 tune=zerolatency byte-stream=true threads=4 ! mpegtsmux ! udpsink host=localhost port=9999"
                , 0, (double)30, cv::Size(640, 480), true);
    if (!writer.isOpened()) {
        printf("=ERR= can't create video writer\n");
        return -1;
    }

    cv::Mat frame;
    int key;

    while (true) {

        cap >> frame;
        if (frame.empty())
            break;

        /* Process the frame here */

        writer << frame;
        key = cv::waitKey( 30 );
    }
}

Hope this helps. ;)

Answer 2

Ok this is long for comment.. its not answer but few hints:

1a, Use netcast to check what is recieved on reciever side.. Its simple:

shell> nc -l 5000 -u

Than check whats being printed when you send something to the reciever.. nc is set to dump everything to the screen..

1b, you can try vlc for reciever and check the debug messages (its located in Tools > Messages - or hit Ctrl+M). Set the log lever to 2 debug .. Then open network stream and use udp://@:5000 as URL..

Btw you could test it with rtp with pipe:

appsrc ! x264enc ! mpegtsmux ! rtpmp2tpay ! udpsink host=localhost port=5000

Which is in vlc rtp://@:5000 then..

2, Check whats flowing after appsrc by using identity element (very helpful.. I use it often to debug pipe problems):

change your pipe to (note the identity element and -v for udpsink):

cv::VideoWriter writer = cv::VideoWriter("appsrc ! identity silent=false ! x264enc ! mpegtsmux ! udpsink -v host=localhost port=5000",  0, (double)30, cv::Size(640, 480), true);

Then run your code and check its output.. it shall list the incomming buffers from appsrc

3, To the code you posted as update - no I meant to use caps= attribute for caps, but maybe there is not difference:

    writer.open("appsrc caps="video/x-raw, framerate=30/1, width=640, height=480, format=RGB" ! autovideoconvert ! ximagesink sync=false", 0, (double)30, cv::Size(640, 480), true);