Pulling data from a CMSampleBuffer in order to create a deep copy

Question

I am trying to create a copy of a CMSampleBuffer as returned by captureOutput in a AVCaptureVideoDataOutputSampleBufferDelegate.

Since the CMSampleBuffers come from a preallocated pool of (15) buffers, if I attach a reference to them they cannot be recollected. This causes all remaining frames to be dropped.

To maintain optimal performance, some sample buffers directly reference pools of memory that may need to be reused by the device system and other capture inputs. This is frequently the case for uncompressed device native capture where memory blocks are copied as little as possible. If multiple sample buffers reference such pools of memory for too long, inputs will no longer be able to copy new samples into memory and those samples will be dropped.

If your application is causing samples to be dropped by retaining the provided CMSampleBufferRef objects for too long, but it needs access to the sample data for a long period of time, consider copying the data into a new buffer and then releasing the sample buffer (if it was previously retained) so that the memory it references can be reused.

Obviously I must copy the CMSampleBuffer but CMSampleBufferCreateCopy() will only create a shallow copy. Thus I conclude that I must use CMSampleBufferCreate(). I filled in the 12! parameters that the constructor needs but ran into the problem that my CMSampleBuffers do not contain a blockBuffer (not entirely sure what that is but it seems important).

This question has been asked several times but not answered.

Deep Copy of CMImageBuffer or CVImageBuffer and Create a copy of CMSampleBuffer in Swift 2.0

One possible answer is "I finally figured out how to use this to create a deep clone. All the copy methods reused the data in the heap which kept would lock the AVCaptureSession. So I had to pull the data out into a NSMutableData object and then created a new sample buffer." credit to Rob on SO. However, I do not know how to do this correcly.

If you are interested, this is the output of print(sampleBuffer). There is no mention of blockBuffer, aka CMSampleBufferGetDataBuffer returns nil. There is a imageBuffer, but creating a "copy" using CMSampleBufferCreateForImageBuffer does not seem to free the CMSampleBuffer either.

---

EDIT: Since this question has been posted I have been trying even more ways of copying the memory.

I did the same thing that user Kametrixom tried. This is my attempt at the same idea, to first copy the CVPixelBuffer then use CMSampleBufferCreateForImageBuffer to create the final sample buffer. However this results in one of two error:

• A EXC_BAD_ACCESS on the memcpy instruction. AKA a segfault from trying to access outside of the application's memory.
• Or, the memory will copy successfully but the CMSampleBufferCreateReadyWithImageBuffer() will fail with result code -12743 which "Indicates that the format of the given media does not match the given format description. For example, a format description paired with a CVImageBuffer that fails CMVideoFormatDescriptionMatchesImageBuffer."

You can see that both Kametrixom and I did use CMSampleBufferGetFormatDescription(sampleBuffer) to try to copy the source buffer's format description. Thus, I'm not sure why the format of the given media does not match the given format description.

Answer 1

Alright, I think I finally got it. I created a helper extension to make a full copy of a CVPixelBuffer:

extension CVPixelBuffer {
    func copy() -> CVPixelBuffer {
        precondition(CFGetTypeID(self) == CVPixelBufferGetTypeID(), "copy() cannot be called on a non-CVPixelBuffer")

        var _copy : CVPixelBuffer?
        CVPixelBufferCreate(
            nil,
            CVPixelBufferGetWidth(self),
            CVPixelBufferGetHeight(self),
            CVPixelBufferGetPixelFormatType(self),
            CVBufferGetAttachments(self, kCVAttachmentMode_ShouldPropagate)?.takeUnretainedValue(),
            &_copy)

        guard let copy = _copy else { fatalError() }

        CVPixelBufferLockBaseAddress(self, kCVPixelBufferLock_ReadOnly)
        CVPixelBufferLockBaseAddress(copy, 0)

        for plane in 0..<CVPixelBufferGetPlaneCount(self) {
            let dest = CVPixelBufferGetBaseAddressOfPlane(copy, plane)
            let source = CVPixelBufferGetBaseAddressOfPlane(self, plane)
            let height = CVPixelBufferGetHeightOfPlane(self, plane)
            let bytesPerRow = CVPixelBufferGetBytesPerRowOfPlane(self, plane)

            memcpy(dest, source, height * bytesPerRow)
        }

        CVPixelBufferUnlockBaseAddress(copy, 0)
        CVPixelBufferUnlockBaseAddress(self, kCVPixelBufferLock_ReadOnly)

        return copy
    }
}

Now you can use this in your didOutputSampleBuffer method:

guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return }

let copy = pixelBuffer.copy()

toProcess.append(copy)

But be aware, one such pixelBuffer takes up about 3MB of memory (1080p), which means that in 100 frames you got already about 300MB, which is about the point at which the iPhone says STAHP (and crashes).

Note that you don't actually want to copy the CMSampleBuffer since it only really contains a CVPixelBuffer because it's an image.

Answer 2

This is the Swift 3 solution to the top rated answer.

extension CVPixelBuffer {
func copy() -> CVPixelBuffer {
    precondition(CFGetTypeID(self) == CVPixelBufferGetTypeID(), "copy() cannot be called on a non-CVPixelBuffer")

    var _copy : CVPixelBuffer?
    CVPixelBufferCreate(
        kCFAllocatorDefault,
        CVPixelBufferGetWidth(self),
        CVPixelBufferGetHeight(self),
        CVPixelBufferGetPixelFormatType(self),
        nil,
        &_copy)

    guard let copy = _copy else { fatalError() }

    CVPixelBufferLockBaseAddress(self, CVPixelBufferLockFlags.readOnly)
    CVPixelBufferLockBaseAddress(copy, CVPixelBufferLockFlags(rawValue: 0))


    let copyBaseAddress = CVPixelBufferGetBaseAddress(copy)
    let currBaseAddress = CVPixelBufferGetBaseAddress(self)

    memcpy(copyBaseAddress, currBaseAddress, CVPixelBufferGetDataSize(self))

    CVPixelBufferUnlockBaseAddress(copy, CVPixelBufferLockFlags(rawValue: 0))
    CVPixelBufferUnlockBaseAddress(self, CVPixelBufferLockFlags.readOnly)


    return copy
}
}

Answer 3

I believe that with VideoToolbox.framework, you can use VTPixelTransferSession to copy pixel buffers. In fact it's the only thing that this class do.

Reference: https://developer.apple.com/documentation/videotoolbox/vtpixeltransfersession-7cg

Answer 4

I spent a good couple hours trying to get this to work. It turns out both the attachments from the original CVPixelBuffer and the IOSurface options found in PixelBufferMetalCompatibilityKey are necessary.

(FYI, VideoToolbox's PixelTransferSession is macOS-only, sadly.)

Here's what I ended up with. I've left in a few lines at the end that allow you to verify the memcpy by comparing the average colours of both the original and copied CVPixelBuffers. It does slow things down so it should be removed once you're confident your copy() is working as expected. The CIImage.averageColour extension is adapted from this code.

extension CVPixelBuffer {
    func copy() -> CVPixelBuffer {
        precondition(CFGetTypeID(self) == CVPixelBufferGetTypeID(), "copy() cannot be called on a non-CVPixelBuffer")

        let ioSurfaceProps = [
            "IOSurfaceOpenGLESFBOCompatibility": true as CFBoolean,
            "IOSurfaceOpenGLESTextureCompatibility": true as CFBoolean,
            "IOSurfaceCoreAnimationCompatibility": true as CFBoolean
        ] as CFDictionary

        let options = [
            String(kCVPixelBufferMetalCompatibilityKey): true as CFBoolean,
            String(kCVPixelBufferIOSurfacePropertiesKey): ioSurfaceProps
        ] as CFDictionary

        var _copy : CVPixelBuffer?
        CVPixelBufferCreate(
            nil,
            CVPixelBufferGetWidth(self),
            CVPixelBufferGetHeight(self),
            CVPixelBufferGetPixelFormatType(self),
            options,
            &_copy)

        guard let copy = _copy else { fatalError() }

        CVBufferPropagateAttachments(self as CVBuffer, copy as CVBuffer)

        CVPixelBufferLockBaseAddress(self, CVPixelBufferLockFlags.readOnly)
        CVPixelBufferLockBaseAddress(copy, CVPixelBufferLockFlags(rawValue: 0))

        let copyBaseAddress = CVPixelBufferGetBaseAddress(copy)
        let currBaseAddress = CVPixelBufferGetBaseAddress(self)

        memcpy(copyBaseAddress, currBaseAddress, CVPixelBufferGetDataSize(self))

        CVPixelBufferUnlockBaseAddress(copy, CVPixelBufferLockFlags(rawValue: 0))
        CVPixelBufferUnlockBaseAddress(self, CVPixelBufferLockFlags.readOnly)

        // let's make sure they have the same average color
//        let originalImage = CIImage(cvPixelBuffer: self)
//        let copiedImage = CIImage(cvPixelBuffer: copy)
//
//        let averageColorOriginal = originalImage.averageColour()
//        let averageColorCopy = copiedImage.averageColour()
//
//        assert(averageColorCopy == averageColorOriginal)
//        debugPrint("average frame color: \(averageColorCopy)")

        return copy
    }
}