Duet - Merge 2 Videos Side by Side

Question

NOTE:- Merge Videos Side By Side WITHOUT Losing Video Quality

I think that is a Very Very Important Question, After a lot of searches & Googling, didn't find any helpful material related to this Question.

I'm working on a Project where I need to MERGE Videos Side-By-Side in a single file.

I had done Merged Videos using AVFoundation But the problem is FIRST Video is showing as an Overlay to a SECOND video(not Merging properly as same as SMULE App/Karaoke App or Tiktok App).

func mergeVideosFilesWithUrl(savedVideoUrl: URL, newVideoUrl: URL, audioUrl:URL)
    {
        let savePathUrl : NSURL = NSURL(fileURLWithPath: NSHomeDirectory() + "/Documents/camRecordedVideo.mp4")
        do { // delete old video
            try FileManager.default.removeItem(at: savePathUrl as URL)
        } catch { print(error.localizedDescription) }

        var mutableVideoComposition : AVMutableVideoComposition = AVMutableVideoComposition()
        var mixComposition : AVMutableComposition = AVMutableComposition()

        let aNewVideoAsset : AVAsset = AVAsset(url: newVideoUrl)
        let asavedVideoAsset : AVAsset = AVAsset(url: savedVideoUrl)

        let aNewVideoTrack : AVAssetTrack = aNewVideoAsset.tracks(withMediaType: AVMediaType.video)[0]
        let aSavedVideoTrack : AVAssetTrack = asavedVideoAsset.tracks(withMediaType: AVMediaType.video)[0]

        let mutableCompositionNewVideoTrack : AVMutableCompositionTrack = mixComposition.addMutableTrack(withMediaType: AVMediaType.video, preferredTrackID: kCMPersistentTrackID_Invalid)!
        do{
            try mutableCompositionNewVideoTrack.insertTimeRange(CMTimeRangeMake(start: CMTime.zero, duration: aNewVideoAsset.duration), of: aNewVideoTrack, at: CMTime.zero)
        }catch {  print("Mutable Error") }

        let mutableCompositionSavedVideoTrack : AVMutableCompositionTrack = mixComposition.addMutableTrack(withMediaType: AVMediaType.video, preferredTrackID: kCMPersistentTrackID_Invalid)!
        do{
            try mutableCompositionSavedVideoTrack.insertTimeRange(CMTimeRangeMake(start: CMTime.zero, duration: asavedVideoAsset.duration), of: aSavedVideoTrack , at: CMTime.zero)
        }catch{ print("Mutable Error") }

        let mainInstruction = AVMutableVideoCompositionInstruction()
        mainInstruction.timeRange = CMTimeRangeMake(start: CMTime.zero, duration: CMTimeMaximum(aNewVideoAsset.duration, asavedVideoAsset.duration) )

        let newVideoLayerInstruction = AVMutableVideoCompositionLayerInstruction(assetTrack: mutableCompositionNewVideoTrack)
        let newScale : CGAffineTransform = CGAffineTransform.init(scaleX: 0.7, y: 0.7)
        let newMove : CGAffineTransform = CGAffineTransform.init(translationX: 230, y: 230)
        newVideoLayerInstruction.setTransform(newScale.concatenating(newMove), at: CMTime.zero)

        let savedVideoLayerInstruction = AVMutableVideoCompositionLayerInstruction(assetTrack: mutableCompositionSavedVideoTrack)
        let savedScale : CGAffineTransform = CGAffineTransform.init(scaleX: 1.2, y: 1.5)
        let savedMove : CGAffineTransform = CGAffineTransform.init(translationX: 0, y: 0)
        savedVideoLayerInstruction.setTransform(savedScale.concatenating(savedMove), at: CMTime.zero)

        mainInstruction.layerInstructions = [newVideoLayerInstruction, savedVideoLayerInstruction]


        mutableVideoComposition.instructions = [mainInstruction]
        mutableVideoComposition.frameDuration = CMTimeMake(value: 1, timescale: 30)
        mutableVideoComposition.renderSize = CGSize(width: 1240 , height: self.camPreview.frame.height)

        finalPath = savePathUrl.absoluteString
        let assetExport: AVAssetExportSession = AVAssetExportSession(asset: mixComposition, presetName: AVAssetExportPresetHighestQuality)!
        assetExport.videoComposition = mutableVideoComposition
        assetExport.outputFileType = AVFileType.mov

        assetExport.outputURL = savePathUrl as URL
        assetExport.shouldOptimizeForNetworkUse = true

        assetExport.exportAsynchronously { () -> Void in
            switch assetExport.status {

            case AVAssetExportSession.Status.completed:
                print("success")
            case  AVAssetExportSession.Status.failed:
                print("failed \(assetExport.error)")
            case AVAssetExportSession.Status.cancelled:
                print("cancelled \(assetExport.error)")
            default:
                print("complete")
            }
        }

    }

And this is my Output

And what I want

As I don't know what is the best approach to make a SIDE BY SIDE VIDEO/DUET VIDEO... As for now, I have used AVFoundation. I have not used any 3rd party framework or any POD.

I would like to ask, what is the BEST Approach to implement this? Videos should Merge on Server side or an App? Also which Approach I should use?

Any help would be highly highly highly appreciated. Thanks

Answer 1

To achieve this, I would create a new AVMutableComposition object containing 2 tracks, and set transform on each to place them side by side:

let composition = AVMutableComposition(urlAssetInitializationOptions: <your options>)
let videoTrackA = composition.addMutableTrack(withMediaType:.video, preferredTrackID:kCMPersistentTrackID_Invalid);
let videoTrackB = composition.addMutableTrack(withMediaType:.video, preferredTrackID:kCMPersistentTrackID_Invalid);

videoTrackA.preferredTransform = CGAffineTransform(translationX: <yourX_for_A>, y:0.0)
videoTrackB.preferredTransform = CGAffineTransform(translationX: <yourX_for_B>, y:0.0)

Then. save it using:

let exporter = AVAssetExportSession(asset:<yourAsset>, presetName:<yourPresetName>)
exporter.exportAsynchronously(completionHandler: <yourCompletionHandler>)

(Swift code not tested).

Answer 2

in fact, AVAssetExportSession is for simple needs, and it is too simple for your situation.

You must use AVAssetWriter.

You add AVAssetWriterInput to your AVAssetWriter.

You can configure trasnform of the AVAssetWriterInput using its transform property.

Then, you feed your AVAssetWriterInput with CMSampleBuffer (each images buffer) using append calls.

See full Apple documentation for detailed example: https://developer.apple.com/library/archive/documentation/AudioVideo/Conceptual/AVFoundationPG/Articles/05_Export.html#//apple_ref/doc/uid/TP40010188-CH9-SW2