How to create a IDirect3DSurface from an IBuffer or byte array in UWP

Question

I want to create a video from a few RenderTargetBitmaps in UWP. I am doing that by using MediaClips. From RenderTargetBitmap i can get an IBuffer or byte array of pixels. To create a MediaClip I need either an image file or an IDirect3DSurface. Creating an image just to create a clip is very expensive, so I thought of using IDirect3DSurface. How can I do this? I have tried this:

        RenderTargetBitmap renderTargetBitmap = new RenderTargetBitmap();
        await renderTargetBitmap.RenderAsync(RenderedGrid, 100, 100);

        IBuffer pixels = await renderTargetBitmap.GetPixelsAsync();
        var values = Enum.GetValues(typeof(DirectXPixelFormat));
        CanvasBitmap bitmap=null;

        foreach (DirectXPixelFormat format in values)
        {
            try
            {
                videoClip = new MediaComposition();
                bitmap = CanvasBitmap.CreateFromBytes(myWidget.Device, pixels, renderTargetBitmap.PixelWidth, renderTargetBitmap.PixelHeight, format);
                StorageFile video2 = await storageFolder.CreateFileAsync("video2" + ".mp4", CreationCollisionOption.ReplaceExisting);
                MediaClip d = MediaClip.CreateFromSurface(bitmap, DateTime.Now - previousFrame+new TimeSpan(100));
                videoClip.Clips.Add(d);
                await videoClip.RenderToFileAsync(video2);
                break;
            }
            catch(Exception e)
            {

            }
        }

I try all the formats in DirectXPixelFormat but none works.

I have a CanvasControl named myWidget that is empty.

I create a CanvasBitmap from Ibuffer (CanvasBitmap implements IDirect3DSurface)

Create a Mediaclip from CanvasBitmap

Add it to MediaComposition.

Then I try to render to video file.When i try to save to a file it throws an error

System.Runtime.InteropServices.COMException Stream is not in a state to handle the request.

EDIT: I figured out where the problem is, but not why and not how to fix it.

            await videoClip.SaveAsync(video2);
            videoClip= await MediaComposition.LoadAsync(video2);
            var x=await videoClip.RenderToFileAsync(video2);

Now with these three lines i can save the video, but using only the third line it throws the error above. I cannot make sense of it. Why does saving and loading fix the problem??

Answer 1

The MediaComposition.RenderToFileAsync Method saves the composition to a video file that can be played back with standard media players. From the error info, it seems the stream content is not correct media data and can not be render into a video file directly.

So, to create a video from a few RenderTargetBitmaps in UWP, the way to use an image file should be your choice. using MediaClip.CreateFromImageFileAsync method by saving the RenderTargetBitmap into a file then using it to create a video.

private async void CreateVideoByConvertRenderBitmapToFile()
{
    var folder = await ApplicationData.Current.LocalFolder.CreateFolderAsync("Test", 
        CreationCollisionOption.ReplaceExisting);
    var composition = new MediaComposition();
    for (int i = 0; i < 5; i++)
    {
        RenderTargetBitmap render = new RenderTargetBitmap();
        await render.RenderAsync(RenderGrid);
        MyImage.Source = render;
        var pixel = await render.GetPixelsAsync();
        var file = await folder.CreateFileAsync("test.png", CreationCollisionOption.GenerateUniqueName);
        using (IRandomAccessStream stream = await file.OpenAsync(FileAccessMode.ReadWrite))
        {
            var logicalDpi = DisplayInformation.GetForCurrentView().LogicalDpi;
            var encoder = await BitmapEncoder.CreateAsync(BitmapEncoder.PngEncoderId, stream);
            encoder.SetPixelData(
                BitmapPixelFormat.Bgra8,
                BitmapAlphaMode.Ignore,
                (uint)render.PixelWidth,
                (uint)render.PixelHeight,
                logicalDpi,
                logicalDpi,
                pixel.ToArray());
            await encoder.FlushAsync();
            stream.Dispose();
            MediaClip clip = await MediaClip.CreateFromImageFileAsync(file, TimeSpan.FromSeconds(3));
            composition.Clips.Add(clip);
            MyText.Text = "First frame >>>" + i;
        }
    }
    var video = await ApplicationData.Current.LocalFolder.CreateFileAsync("test.mp4",
        CreationCollisionOption.ReplaceExisting);
    var action = await composition.RenderToFileAsync(video, MediaTrimmingPreference.Precise);
    await folder.DeleteAsync();
}

Answer 2

Most probable reason is that, CanvasBitmap has an underlying IDirce3DSurface object as well as image data like byte[] or something else , though I am not sure about this.

If that's true, then creating a CanvasBitmap from byte[] or IBuffer won't effect the underlying IDirect3DSurface, the image part will be constructed only. You can see that by saving that image on the disk, it gives no error.

But I think there's a workaround if you want to skip saving data on the disk:

You can contruct the underlying IDirect3DSurface if you do Offscreen Drawing to a CanvasRenderTarget.

So, you can use the CanvasBitmap to construct a CanvasRenderTarget and then use that CanvasRenderTarget to contruct a MediaClip:

CanvasRenderTarget rendertarget;
using (CanvasBitmap canvas = CanvasBitmap.CreateFromBytes(CanvasDevice.GetSharedDevice(), pixels, renderTargetBitmap.PixelWidth, renderTargetBitmap.PixelHeight, format))
{
     rendertarget = new CanvasRenderTarget(CanvasDevice.GetSharedDevice(), canvas.SizeInPixels.Width, canvas.SizeInPixels.Height, 96);
     using (CanvasDrawingSession ds = rendertarget.CreateDrawingSession())
     {
         ds.Clear(Colors.Black);
         ds.DrawImage(canvas);
     }
}
MediaClip d = MediaClip.CreateFromSurface(renderTarget, TimeSpan.FromMilliseconds(80));
mc.Clips.Add(m);