How to generate an UIImage from AVCapturePhoto with correct orientation?

Question

I am calling AVFoundation\'s delegate method to handle a photo capture, but I am having difficulty converting the AVCapturePhoto it generates into

Answer 1

Final update: I ran some experiments with the app and came to the following conclusions:

1. kCGImagePropertyOrientation doesn’t seem to influence the orientation of the captured image inside your application and it only varies with the device orientation if you update your photoOutput connection each time you are about to call the capturePhoto method. So:

   func snapPhoto() {
       // prepare and initiate image capture routine
   
       // if I leave the next 4 lines commented, the intented orientation of the image on display will be 6 (right top) - kCGImagePropertyOrientation
       let deviceOrientation = UIDevice.current.orientation // retrieve current orientation from the device
       guard let photoOutputConnection = capturePhotoOutput.connection(with: AVMediaType.video) else {fatalError("Unable to establish input>output connection")}// setup a connection that manages input > output
       guard let videoOrientation = deviceOrientation.getAVCaptureVideoOrientationFromDevice() else {return}
       photoOutputConnection.videoOrientation = videoOrientation // update photo's output connection to match device's orientation
   
       let photoSettings = AVCapturePhotoSettings()
       photoSettings.isAutoStillImageStabilizationEnabled = true
       photoSettings.isHighResolutionPhotoEnabled = true
       photoSettings.flashMode = .auto
       self.capturePhotoOutput.capturePhoto(with: photoSettings, delegate: self) // trigger image capture. It appears to work only if the capture session is running.
   }
   

2. Viewing the generated images on the debugger has shown me how they get generated, so I could infer the required rotation (UIImageOrientation) to get it displayed upright. In other words: updating UIImageOrientation tells how the image should be rotated in order for you to see it in the correct orientation. So I came to the following table:

3. I had to update my UIDeviceOrientation extension to a rather unintuitive form:

   extension UIDeviceOrientation {
       func getUIImageOrientationFromDevice() -> UIImageOrientation {
           // return CGImagePropertyOrientation based on Device Orientation
           // This extented function has been determined based on experimentation with how an UIImage gets displayed.
           switch self {
           case UIDeviceOrientation.portrait, .faceUp: return UIImageOrientation.right
           case UIDeviceOrientation.portraitUpsideDown, .faceDown: return UIImageOrientation.left
           case UIDeviceOrientation.landscapeLeft: return UIImageOrientation.up // this is the base orientation
           case UIDeviceOrientation.landscapeRight: return UIImageOrientation.down
           case UIDeviceOrientation.unknown: return UIImageOrientation.up
           }
       }
   }
   

4. This is how my final delegate method looks now. It displays the image in the expected orientation.

   func photoOutput(_ output: AVCapturePhotoOutput,
                                    didFinishProcessingPhoto photo: AVCapturePhoto,
                                    error: Error?)
   {
       // capture image finished
       print("Image captured.")
   
       let photoMetadata = photo.metadata
       // Returns corresponting NSCFNumber. It seems to specify the origin of the image
       //                print("Metadata orientation: ",photoMetadata["Orientation"])
   
       // Returns corresponting NSCFNumber. It seems to specify the origin of the image
       print("Metadata orientation with key: ",photoMetadata[String(kCGImagePropertyOrientation)] as Any)
   
       guard let imageData = photo.fileDataRepresentation() else {
           print("Error while generating image from photo capture data.");
           self.lastPhoto = nil; self.controller.goToProcessing();
           return
   
       }
   
       guard let uiImage = UIImage(data: imageData) else {
           print("Unable to generate UIImage from image data.");
           self.lastPhoto = nil; self.controller.goToProcessing();
           return
   
       }
   
       // generate a corresponding CGImage
       guard let cgImage = uiImage.cgImage else {
           print("Error generating CGImage");self.lastPhoto=nil;return
   
       }
   
       guard let deviceOrientationOnCapture = self.deviceOrientationOnCapture else {
           print("Error retrieving orientation on capture");self.lastPhoto=nil;
           return
   
       }
   
       self.lastPhoto = UIImage(cgImage: cgImage, scale: 1.0, orientation: deviceOrientationOnCapture.getUIImageOrientationFromDevice())
   
       print(self.lastPhoto)
       print("UIImage generated. Orientation:(self.lastPhoto.imageOrientation.rawValue)")
       self.controller.goToProcessing()
   }
   
   
   func photoOutput(_ output: AVCapturePhotoOutput, 
                      willBeginCaptureFor resolvedSettings: AVCaptureResolvedPhotoSettings) 
                      {
       print("Just about to take a photo.")
       // get device orientation on capture
       self.deviceOrientationOnCapture = UIDevice.current.orientation
       print("Device orientation: \(self.deviceOrientationOnCapture.rawValue)")
   }