Best path from AVPlayerItemVideoOutput to openGL Texture

Question

Been pulling my hair out trying to figure out the current best path from AVFoundation videos to an openGLTexture, most of what I find is related to iOS, and I can't seem to make it work well in OSX.

First of all, this is how I set up the videoOutput:

NSDictionary *pbOptions = [NSDictionary dictionaryWithObjectsAndKeys:
                          [NSNumber numberWithInt:kCVPixelFormatType_422YpCbCr8], kCVPixelBufferPixelFormatTypeKey,
                          [NSDictionary dictionary], kCVPixelBufferIOSurfacePropertiesKey,
                                   nil];
        self.playeroutput   = [[AVPlayerItemVideoOutput alloc] initWithPixelBufferAttributes:pbOptions];
        self.playeroutput.suppressesPlayerRendering = YES;

I'm attempting three different solutions, of which only one seems to work consistently, but not sure it's the fastest. One works for a little, then breaks down with frames jumping all over the place, and one just produces black.

First off, working solution using glTexImage2D

- (BOOL) renderWithCVPixelBufferForTime: (NSTimeInterval) time
{
    CMTime vTime = [self.playeroutput itemTimeForHostTime:CACurrentMediaTime()];

    if ([self.playeroutput hasNewPixelBufferForItemTime:vTime]) {
        if (_cvPixelBufferRef) {
            CVPixelBufferUnlockBaseAddress(_cvPixelBufferRef, kCVPixelBufferLock_ReadOnly);
            CVPixelBufferRelease(_cvPixelBufferRef);
        }
        _cvPixelBufferRef = [self.playeroutput copyPixelBufferForItemTime:vTime itemTimeForDisplay:NULL];

        CVPixelBufferLockBaseAddress(_cvPixelBufferRef, kCVPixelBufferLock_ReadOnly);
        GLsizei       texWidth    = CVPixelBufferGetWidth(_cvPixelBufferRef);
        GLsizei       texHeight   = CVPixelBufferGetHeight(_cvPixelBufferRef);
        GLvoid *baseAddress = CVPixelBufferGetBaseAddress(_cvPixelBufferRef);


        glBindTexture(GL_TEXTURE_RECTANGLE_ARB, self.textureName);
        glTexParameteri(GL_TEXTURE_RECTANGLE_ARB, GL_TEXTURE_STORAGE_HINT_APPLE , GL_STORAGE_CACHED_APPLE);
        glTexImage2D(GL_TEXTURE_RECTANGLE_ARB, 0, GL_RGB, texWidth, texHeight, 0, GL_YCBCR_422_APPLE, GL_UNSIGNED_SHORT_8_8_APPLE, baseAddress);

        glBindTexture(GL_TEXTURE_RECTANGLE_ARB, 0);
    } 
    return YES;
} 

This method spends most of it's time Locking the base address of the pixel buffer, but the docs say it's not required if accessing data from the GPU and can impair performance. I could not figure out a way to get a texture without locking.

Next up, the almost working solution using iOSurface, this works for a bit then gets really glitchy, as if ioSurfaces are being used from previous frames:

- (BOOL) renderWithIOSurfaceForTime:(NSTimeInterval) time {
    CMTime vTime = [self.playeroutput itemTimeForHostTime:CACurrentMediaTime()];

    if ([self.playeroutput hasNewPixelBufferForItemTime:vTime]) {

         CVPixelBufferRef pb = [self.playeroutput copyPixelBufferForItemTime:vTime itemTimeForDisplay:NULL];
         IOSurfaceRef newSurface = CVPixelBufferGetIOSurface(pb);
         if (_surfaceRef != newSurface) {
            IOSurfaceDecrementUseCount(_surfaceRef);
            _surfaceRef = newSurface;
            IOSurfaceIncrementUseCount(_surfaceRef);
            GLsizei       texWidth = (int) IOSurfaceGetWidth(_surfaceRef);
            GLsizei       texHeight= (int) IOSurfaceGetHeight(_surfaceRef);
            size_t        rowbytes = CVPixelBufferGetBytesPerRow(_cvPixelBufferRef);

            glBindTexture(GL_TEXTURE_RECTANGLE_ARB, self.textureName);
            CGLTexImageIOSurface2D(cgl_ctx, GL_TEXTURE_RECTANGLE_ARB, GL_RGB8, texWidth, texHeight, GL_YCBCR_422_APPLE, GL_UNSIGNED_SHORT_8_8_APPLE, _surfaceRef, 0);
            glBindTexture(GL_TEXTURE_RECTANGLE_ARB, 0);
        }
        CVPixelBufferRelease(pb);   
    }
    return YES;
}

This seems to be the best solution, if it would work. I have another process that creates textures from ioSurfaces and it works just fine, while being extremely fast too.

Finally the one that seems recommended for iOS is to use a CVOpenGLTextureCache, implementation in osx seems slightly different, and I could not get it to render anything but black, plus it seemed even slower than the first solution....

- (BOOL) renderByCVOpenGLTextureCacheForTime:(NSTimeInterval) time
{
    CMTime vTime = [self.playeroutput itemTimeForHostTime:CACurrentMediaTime()];

    if ([self.playeroutput hasNewPixelBufferForItemTime:vTime]) {
        _cvPixelBufferRef = [self.playeroutput copyPixelBufferForItemTime:vTime itemTimeForDisplay:NULL];
        if (!_textureCacheRef) {
            CVReturn error = CVOpenGLTextureCacheCreate(kCFAllocatorDefault, NULL, cgl_ctx, CGLGetPixelFormat(cgl_ctx), NULL, &_textureCacheRef);
            if (error) {
                NSLog(@"Texture cache create failed");
            }
        }

        CVReturn error = CVOpenGLTextureCacheCreateTextureFromImage(kCFAllocatorDefault, _textureCacheRef, _cvPixelBufferRef, NULL, &_textureRef);
        if (error) {
            NSLog(@"Failed to copy video texture");
        }


        CVPixelBufferRelease(_cvPixelBufferRef);

        _textureName = CVOpenGLTextureGetName(_textureRef);

    }
    return YES;
}

Probably I'm not setting things up right, there's zero documentation for the texture cache in osx.

I've found it best to retain the cvpixelbufferref between render cycles, as I understand it, the texture upload can run asynchronously with CGLTexImage2d, I'm quite happy with that, several other objects may be rendered at the same time, a cglflushDrawable is eventually called when textures are eventually drawn.

Most of the apple examples I find for video to openGL Textures relate to iOS, and split the texture in two to recombine in a shader, like in this example https://developer.apple.com/library/ios/samplecode/GLCameraRipple/Listings/GLCameraRipple_main_m.html#//apple_ref/doc/uid/DTS40011222-GLCameraRipple_main_m-DontLinkElementID_11 I couldn't adapt the code directly as the texture cache has different implementations in iOS.

So any pointers would be great, it seems like vital functionality, but info I find regarding av foundation and opengl on osx seems very negative.

Update: Updated ioSurface code with use counts, works slightly longer, but still glitches out eventually.

Answer 1

I'm starting on the same journey and know as much about OpenGL as I do about sheep farming, but did notice that your pbOptions doesn't contain kCVPixelBufferOpenGLCompatibilityKey

NSDictionary *pbOptions = [NSDictionary dictionaryWithObjectsAndKeys:
    [NSNumber numberWithInt:kCVPixelFormatType_422YpCbCr8], kCVPixelBufferPixelFormatTypeKey,
    [NSDictionary dictionary], kCVPixelBufferIOSurfacePropertiesKey,
    [NSNumber numberWithBool:YES], kCVPixelBufferOpenGLCompatibilityKey, nil];

I'm requesting the pixel buffer as kCVPixelFormatType_32BGRA rather than component and this works for me with local variables for _currentSurface (IOSurfaceRef), textureName (GLuint), _sourceWidth (int) and _sourceHeight (int)

IOSurfaceRef newSurface = CVPixelBufferGetIOSurface(pixelBuffer);
if (_currentSurface != newSurface) {
    CGLContextObj  cgl_ctx = (CGLContextObj)[[self openGLContext] CGLContextObj];
    [[self openGLContext] makeCurrentContext];

    IOSurfaceDecrementUseCount(_currentSurface);
    _currentSurface = newSurface;
    IOSurfaceIncrementUseCount(_currentSurface);
    GLsizei texWidth = (int) IOSurfaceGetWidth(_currentSurface);
    GLsizei texHeight = (int) IOSurfaceGetHeight(_currentSurface);

    if (_sourceWidth == 0 && _sourceHeight == 0) {
        // used during drawing of texture
        _sourceWidth = texWidth;
        _sourceHeight = texHeight;
    }

    if (!_textureName) {
        GLuint name;
        glGenTextures(1, &name);
        _textureName = name;
    }

    glBindTexture(GL_TEXTURE_RECTANGLE_ARB, _textureName);
    CGLTexImageIOSurface2D(cgl_ctx, GL_TEXTURE_RECTANGLE_ARB, GL_RGBA, texWidth, texHeight, GL_BGRA, GL_UNSIGNED_INT_8_8_8_8_REV, self.currentSurface, 0);        
    glBindTexture(GL_TEXTURE_RECTANGLE_ARB, 0);
}

Answer 2

When using option #2, did you set the "retains backing" property of the EAGLLayer to NO? That may be why it appears to be using the same frame in different cycles. It would be beneficial to see how you configured, or whether you configured, the layer appropriately for the view to which you're rendering framebuffers...

Answer 3

It appears that the corresponding method of fast video texturing on OSX is to use BiPlanar IOSurfaces where surfacePlane[0] is the luma (Y) and surfacePlane[1] is the subsampled chroma (UV). My code runs on Core Profile, so the GLenum constants to CGLTexImageIOSurface2D reflect the same. Pretty sure that only rectangle textures are supported. I use a GLSL shader to combine them and it's working great on Sierra. Briefly summarized:

NSDictionary* pixelBufferAttributesIOSurfaceBiPlanar = [NSDictionary dictionaryWithObjectsAndKeys:
[NSNumber numberWithUnsignedInt:kPixelFormatTypeGL], ARC_BRIDGE(id)kCVPixelBufferPixelFormatTypeKey,
[NSNumber numberWithBool:YES], ARC_BRIDGE(id)kCVPixelBufferOpenGLCompatibilityKey,
[NSNumber numberWithBool:YES], ARC_BRIDGE(id)kCVPixelBufferIOSurfaceOpenGLTextureCompatibilityKey,
[NSDictionary dictionary], ARC_BRIDGE(id)kCVPixelBufferIOSurfacePropertiesKey,
nil];

/* luma texture */
CGLTexImageIOSurface2D(glContext,GL_TEXTURE_RECTANGLE,GL_R8,(GLsizei)textureSize.width,(GLsizei)textureSize.height,GL_RED,GL_UNSIGNED_BYTE,surfaceRef,0);
/* chroma texture */
CGLTexImageIOSurface2D(glContext,GL_TEXTURE_RECTANGLE,GL_RG8,(GLsizei)textureSize.width,(GLsizei)textureSize.height,GL_RG,GL_UNSIGNED_BYTE,surfaceRef,1);

GLSL

uniform sampler2DRect textureSampler0;
uniform sampler2DRect textureSampler1;
// ...
vec3 YCrCb;
vec2 lumaCoords = texCoord0;
vec2 chromaCoords = lumaCoords*vec2(0.5,0.5);
vec2 chroma = texture(textureSampler1,chromaCoords).xy;
float luma = texture(textureSampler0,lumaCoords).x;
YCrCb.x = (luma-(16.0/255.0)); // video range
YCrCb.yz = (chroma-vec2(0.5,0.5));
vec4 rgbA = vec4(colorConversionMatrix * YCrCb,1.0);

The color conversion matrix should be generated from the CVPixelBufferRef

CFTypeRef colorAttachments = CVBufferGetAttachment(pixelBuffer, kCVImageBufferYCbCrMatrixKey, NULL);

IOSurfaceRef is a CFTypeRef derived object and I use CFRetain/CFRelease to hold onto the surface until I don't need the texture. If CGLTexImageIOSurface2D does a GPU blit to upload the texture data, probably only need to CFRetain/Release around the call to CGLTexImageIOSurface2D.

I started with the Apple iOS sample code AVBasicVideoOutput. I ported it to Metal for both iOS & OSX in two days and then spent a week trying to figure out the OpenGL Core Profile version. If you can, use Metal: it's faster and the code is almost exactly the same on both iOS/OSX. Also, there is some good information in WWDC13 Session 507 What's New in OpenGL for OS X. In particular, that there is a GLSL shader compatibility flag that allows EAGL shaders to run mostly unmodified on OSX.