Can I apply a CIFilter to ARkit camera feed?

问题

I'm trying to apply a blur effect to camera live steam image in ARSCNView. I have checked the WWDC videos. They only mentioned the custom rendering with Metal, but I didn't found any complete example on web. Any idea how to do that?

Updated 1 I have tried to apply a filter to the background. It show incorrect orientation. How can I fix this?

let bg=self.session.currentFrame?.capturedImage

        if(bg != nil){
            let context = CIContext()
            let filter:CIFilter=CIFilter(name:"CIColorInvert")!
            let image:CIImage=CIImage(cvPixelBuffer: bg!)
            filter.setValue(image, forKey: kCIInputImageKey)
            let result=filter.outputImage!
            self.sceneView.scene.background.contents = context.createCGImage(result, from: result.extent)

        }

回答1:

I've found a pretty good solution, which is to simply apply a corresponding geometric transform to the background property whenever the device orientation changes:

func session(_ session: ARSession, didUpdate frame: ARFrame) {
    let image = CIImage(cvPixelBuffer: frame.capturedImage)
    filter.setValue(image, forKey: kCIInputImageKey)

    let context = CIContext()
    if let result = filter.outputImage,
        let cgImage = context.createCGImage(result, from: result.extent) {

        sceneView.scene.background.contents = cgImage
        if let transform = currentScreenTransform() {
            sceneView.scene.background.contentsTransform = transform
        }
    }
}

private func currentScreenTransform() -> SCNMatrix4? {
    switch UIDevice.current.orientation {
    case .landscapeLeft:
        return SCNMatrix4Identity
    case .landscapeRight:
        return SCNMatrix4MakeRotation(.pi, 0, 0, 1)
    case .portrait:
        return SCNMatrix4MakeRotation(.pi / 2, 0, 0, 1)
    case .portraitUpsideDown:
        return SCNMatrix4MakeRotation(-.pi / 2, 0, 0, 1)
    default:
        return nil
    }
}

Make sure you call UIDevice.current.beginGeneratingDeviceOrientationNotifications() in your viewDidLoad method first.

回答2:

I had the same problem. You can try this

let bg = sceneView.session.currentFrame?.capturedImage
    if (bg != nil) {
        let context = CIContext()
        let filter:CIFilter = CIFilter(name: "CIColorInvert")!

        let image:CIImage = CIImage(cvPixelBuffer: bg!, options: nil)

        let cgImage:CGImage = context.createCGImage(image, from: image.extent)!
        let uiImage:UIImage = UIImage.init(cgImage: cgImage)
        let resultImage = CIImage(image: uiImage)?.oriented(forExifOrientation: imageOrientationToDeviceOrientation(value: UIDevice.current.orientation))

        filter.setValue(resultImage, forKey: kCIInputImageKey)

        let result = filter.outputImage!
        self.sceneView.scene.background.contents = context.createCGImage(result, from: result.extent)
    }


func imageOrientationToDeviceOrientation(value: UIDeviceOrientation) -> Int32 {
            switch (value) {
            case .portrait:
                return 6
            case .landscapeLeft:
                return 1
            case .landscapeRight:
                return 3
            default:
                return 6
            }
        }

来源：https://stackoverflow.com/questions/44624937/can-i-apply-a-cifilter-to-arkit-camera-feed

标签

cifilter

ios11

arkit