Perform live video stream processing from CaptureElement & MediaCapture

Question

In my Windows 8 Store Application, I tend to perform live video stream processing (Face detection for example).

---

Technique 1

In my previous Windows 7 application, I was able to use the following technique (Camera Face Detection in C# Using Emgu CV (OpenCV in .NET) and WPF) to perform live video processing. The technique was, having a fixed period timer callback, to actively query image buffer from camera object.

void timer_Tick(object sender, EventArgs e)
{
    Image<Bgr,Byte> currentFrame = capture.QueryFrame();
    // Perform image processing based on currentFrame

---

Technique 2

Another technique I had used before in Android is that, I will install a camera buffer preview callback on camera itself. The camera will periodically trigger the callback, by passing along the captured camera buffer. From there, I can perform image processing.

public abstract void onPreviewFrame (byte[] data, Camera camera)
{
    // Perform image processing based on data

---

However, in articles which talks about video processing for Windows 8 Store Application, they are mostly using built-in processing functions

• How to rotate captured video (Windows Store apps using C#/VB/C++ and XAML)
• How to add video stabilization effect to captured video (Windows Store apps using C#/VB/C++ and XAML)

None of them demonstrate how to access raw camera captured buffer, iterate the buffer pixel by pixel.

I believe I need to make use of CaptureElement & MediaCapture according to

http://blog.xyzzer.me/2012/01/22/displaying-webcam-preview-in-a-metro-app-using-winrt-and-xaml/

This method is great if you just want to see the webcam input or capture it to a file with no hassle. If you want to process the video in real time or overlay some other UI components – enter…

The CaptureElement & MediaCapture Way

However, the author doesn't elaborate more after that. (I hope he does so :)

Any hint how to perform live video stream processing from CaptureElement & MediaCapture?

Answer 1

What you could try first is MediaCapture.CapturePhotoToStreamAsync which I think also only scenarios where that photo will be compressed, but you could probably fairly quickly decode it back into a WriteableBitmap and do some processing on its pixels. You probably aren't going to process too many frames per second that way though.

The better way to do it would probably be to use MediaCapture.AddEffectAsync() and write a Media Foundation Transform (MFT) in C++ that would process the video and somehow communicate the results back to the app.