Fixing orientation when stitching (merging) videos using AVMutableComposition

18
votes

TLDR - SEE EDIT

I am creating a test app in Swift where I want to stitch multiple videos together from my apps documents directory using AVMutableComposition.

I have had success in doing this to some degree, all my videos are stitched together and everything is showing the correct size portrait and landscape.

My issue is, however, that all the videos are showing in the orientation of the last video in the compilation.

I know that to fix this I will need to add layer instructions for each track I add, however I can't seem to get this right, with the answers I have found the entire compilation seems to come out in a portrait orientation with landscape videos simply scaled to fit in the portrait view, so when I turn my phone on its side to view the landscape videos they are still small since they have been scaled to a portrait size.

This is not the outcome I am looking for, I want the expected functionality i.e. if a video is landscape it shows scaled when in portrait mode but if the phone is rotated I want that landscape video to fill the screen (as it would when simply viewing a landscape video in photos) and the same for portrait so that when viewing in portrait it is full screen and when turned sideways the video is scaled to landscape size (like it does when viewing a portrait video in photos).

In summary the desired outcome I want is that when viewing a compilation that has landscape and portrait videos I can view the entire compilation with my phone on its side and the landscape videos are full screen and portrait is scaled, or when viewing the same video in portrait the portrait videos are full screen and the landscape videos are scaled to size.

With all the answers I found this was not the case and they all seemed to have very unexpected behaviour when importing a video from photos to add to the compilation, and the same random behaviour when adding videos that were shot with the front facing camera (to be clear with my current implementation videos imported from library and "selfie" videos appear at the correct size without these problems).

I'm looking for a way to rotate/scale these videos so that they are always showing in the correct orientation and scale depending on which way round the user is holding their phone.

EDIT: I am now know that i can't have both landscape and portrait orientations in one single video, so the expected outcome I'm looking for would be to have the final video in landscape orientation. i have figured out how to make switch all the orientations and scales to get everything the same way up but my output is a portrait video if anyone could help me change this so my output is landscape it would be appreciated.

Below is my function to get the instruction for each video:

func videoTransformForTrack(asset: AVAsset) -> CGAffineTransform
{
    var return_value:CGAffineTransform?

    let assetTrack = asset.tracksWithMediaType(AVMediaTypeVideo)[0]

    let transform = assetTrack.preferredTransform
    let assetInfo = orientationFromTransform(transform)

    var scaleToFitRatio = UIScreen.mainScreen().bounds.width / assetTrack.naturalSize.width
    if assetInfo.isPortrait
    {
        scaleToFitRatio = UIScreen.mainScreen().bounds.width / assetTrack.naturalSize.height
        let scaleFactor = CGAffineTransformMakeScale(scaleToFitRatio, scaleToFitRatio)
        return_value = CGAffineTransformConcat(assetTrack.preferredTransform, scaleFactor)
    }
    else
    {
        let scaleFactor = CGAffineTransformMakeScale(scaleToFitRatio, scaleToFitRatio)
        var concat = CGAffineTransformConcat(CGAffineTransformConcat(assetTrack.preferredTransform, scaleFactor), CGAffineTransformMakeTranslation(0, UIScreen.mainScreen().bounds.width / 2))
        if assetInfo.orientation == .Down
        {
            let fixUpsideDown = CGAffineTransformMakeRotation(CGFloat(M_PI))
            let windowBounds = UIScreen.mainScreen().bounds
            let yFix = assetTrack.naturalSize.height + windowBounds.height
            let centerFix = CGAffineTransformMakeTranslation(assetTrack.naturalSize.width, yFix)
            concat = CGAffineTransformConcat(CGAffineTransformConcat(fixUpsideDown, centerFix), scaleFactor)
        }
        return_value = concat
    }
    return return_value!
}

And the exporter:

    // Create AVMutableComposition to contain all AVMutableComposition tracks
    let mix_composition = AVMutableComposition()
    var total_time = kCMTimeZero

    // Loop over videos and create tracks, keep incrementing total duration
    let video_track = mix_composition.addMutableTrackWithMediaType(AVMediaTypeVideo, preferredTrackID: CMPersistentTrackID())

    var instruction = AVMutableVideoCompositionLayerInstruction(assetTrack: video_track)
    for video in videos
    {
        let shortened_duration = CMTimeSubtract(video.duration, CMTimeMake(1,10));
        let videoAssetTrack = video.tracksWithMediaType(AVMediaTypeVideo)[0]

        do
        {
            try video_track.insertTimeRange(CMTimeRangeMake(kCMTimeZero, shortened_duration),
                ofTrack: videoAssetTrack ,
                atTime: total_time)

            video_track.preferredTransform = videoAssetTrack.preferredTransform

        }
        catch _
        {
        }

        instruction.setTransform(videoTransformForTrack(video), atTime: total_time)

        // Add video duration to total time
        total_time = CMTimeAdd(total_time, shortened_duration)
    }

    // Create main instrcution for video composition
    let main_instruction = AVMutableVideoCompositionInstruction()
    main_instruction.timeRange = CMTimeRangeMake(kCMTimeZero, total_time)
    main_instruction.layerInstructions = [instruction]
    main_composition.instructions = [main_instruction]
    main_composition.frameDuration = CMTimeMake(1, 30)
    main_composition.renderSize = CGSize(width: UIScreen.mainScreen().bounds.width, height: UIScreen.mainScreen().bounds.height)

    let exporter = AVAssetExportSession(asset: mix_composition, presetName: AVAssetExportPreset640x480)
    exporter!.outputURL = final_url
    exporter!.outputFileType = AVFileTypeMPEG4
    exporter!.shouldOptimizeForNetworkUse = true
    exporter!.videoComposition = main_composition

    // 6 - Perform the Export
    exporter!.exportAsynchronouslyWithCompletionHandler()
    {
        // Assign return values based on success of export
        dispatch_async(dispatch_get_main_queue(), { () -> Void in
                self.exportDidFinish(exporter!)
        })
    }

Sorry for the long explanation I just wanted to make sure I was very clear with what I was asking because other answers have not worked for me.

1
iosswiftvideoavmutablecomposition
in regards to the edit i mentioned i was posting a bounty to let someone know i will award points for a correct answer even if no bounty was posted yet IE within the first 2 days. i have now added a bounty and still gained zero attention on this question except for edits???? – AngryDuck
For a question with a dependency on external data like this one, it would be helpful if you could post the whole project, including example videos, so it's easy to reproduce. – jtbandes
This a complete minimum sample that will show the problem you just need 2 videos as AVAssets in the videos array one on landscape and one in portrait and you will see the problem – AngryDuck
Upon reading again, I'm not sure you can achieve exactly what you want. A video is fixed size; it can't change size during the video. Do you want the result to be a square video which is larger than the screen? – jtbandes
After looking at some other things it would seem you are correct, the intended outcome I would actually like is for the final video to be landscape all examples I find using composition layer instructions end up with a portrait final video. I have code that has achieved this which I will post in a bout an hour when I'm home and then could see if someone could help me change it so that the end video is in landscape orientation instead of portrait – AngryDuck

1 Answers

2
votes

Im not sure your orientationFromTransform() give you the correct orientation.

I think you try to modify it or try anything like:

extension AVAsset {

    func videoOrientation() -> (orientation: UIInterfaceOrientation, device: AVCaptureDevicePosition) {
        var orientation: UIInterfaceOrientation = .Unknown
        var device: AVCaptureDevicePosition = .Unspecified

        let tracks :[AVAssetTrack] = self.tracksWithMediaType(AVMediaTypeVideo)
        if let videoTrack = tracks.first {

            let t = videoTrack.preferredTransform

            if (t.a == 0 && t.b == 1.0 && t.d == 0) {
                orientation = .Portrait

                if t.c == 1.0 {
                    device = .Front
                } else if t.c == -1.0 {
                    device = .Back
                }
            }
            else if (t.a == 0 && t.b == -1.0 && t.d == 0) {
                orientation = .PortraitUpsideDown

                if t.c == -1.0 {
                    device = .Front
                } else if t.c == 1.0 {
                    device = .Back
                }
            }
            else if (t.a == 1.0 && t.b == 0 && t.c == 0) {
                orientation = .LandscapeRight

                if t.d == -1.0 {
                    device = .Front
                } else if t.d == 1.0 {
                    device = .Back
                }
            }
            else if (t.a == -1.0 && t.b == 0 && t.c == 0) {
                orientation = .LandscapeLeft

                if t.d == 1.0 {
                    device = .Front
                } else if t.d == -1.0 {
                    device = .Back
                }
            }
        }

        return (orientation, device)
    }
}