iOS CIFaceDetector very slow with Metal

问题

I've been trying to apply filters to a certain part of the face detected in an image.

In order to apply filters to the whole image, I used the sample code from apple: https://developer.apple.com/documentation/avfoundation/cameras_and_media_capture/avcamfilter_applying_filters_to_a_capture_stream

If I just add one line of detecting faces via CIDetector, to the method which sends out CVPixelBuffers to FilterRenderer class and then to MTKView to render the filtered buffer, the performance is dropped down heavily.

The chain goes like this: CMSampleBuffer > CVImageBuffer > CIImage > Detect Faces > Apply filter to face only > Get the CVPixelBuffer back from FilterRenderer > Send it to MTKView

The "Detect Faces" part is so slow, I can't imagine If I do some more processing (find eye and mouth positions), how slow it would be.

You can see the sample implementation here: https://github.com/nipun0505/FaceDetectionMetal

func processVideo(sampleBuffer: CMSampleBuffer) {
    if !renderingEnabled {
        return
    }

    guard let videoPixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer),
        let formatDescription = CMSampleBufferGetFormatDescription(sampleBuffer) else {
            return
    }

    var finalVideoPixelBuffer = videoPixelBuffer
    if let filter = videoFilter {
        if !filter.isPrepared {
            filter.prepare(with: formatDescription, outputRetainedBufferCountHint: 3)
        }

        //Detect faces
        if let faceDetector = faceDetector{
            let features = faceDetector.features(in: CIImage(cvImageBuffer: videoPixelBuffer))
        }

        // Send the pixel buffer through the filter
        guard let filteredBuffer = filter.render(pixelBuffer: finalVideoPixelBuffer) else {
            print("Unable to filter video buffer")
            return
        }

        finalVideoPixelBuffer = filteredBuffer
    }

    previewView.pixelBuffer = finalVideoPixelBuffer
}

Am I doing something wrong here? I tried the same without MTKView where I just detect faces and overlay some images on the AVCaptureVideoPreviewLayer which went very smooth. Not sure what is slowing down my performance.

来源：https://stackoverflow.com/questions/55062672/ios-cifacedetector-very-slow-with-metal

标签

ios

avfoundation

metal

core-image