Rotating Video with AVMutableVideoCompositionLayerInstruction

Question

I'm shooting video on an iPhone 4 with the front camera and combining the video with some other media assets. I'd like for this video to be portrait orientation - the default orientation for all video is landscape and in some circumstances, you have to manage this manually.

I'm using AVFoundation and specifically AVAssetExportSession with a AVMutableVideoComposition. Based on the WWDC videos, it's clear that I have to handle 'fixing' the orientation myself when I'm combining videos into a new composition.

So, I've created an AVMutableVideoCompositionLayerInstruction attached to my AVMutableVideoCompositionInstruction and I'm using the setTransform:atTime: method to set a transform designed to rotate the video:

    AVMutableVideoCompositionLayerInstruction *passThroughLayer = [AVMutableVideoCompositionLayerInstruction videoCompositionLayerInstructionWithAssetTrack:videoTrack];
CGAffineTransform portraitRotationTransform = CGAffineTransformMakeRotation(degreesToRadians(90.0));
[passThroughLayer setTransform:portraitRotationTransform atTime:kCMTimeZero];

The problem is that when I view the video that is exported, none of the actual contents are on the screen. If I reduce the rotation angle to something like 45 degrees, I can see part of the video on the screen - it's almost as if it's not rotating at the center point. I'm including images below to make it more clear what I'm talking about.

The natural size of the video is coming back as 480x360. I've tried changing that to 360x480 and it doesn't impact the core issue.

0 Degree Rotation:

45 Degree Rotation:

A 90 degree rotation is just all green.

Anyway, I'm hoping someone who has done this before can point me in the right direction. I can't find any docs on some of the more advanced topics in AVFoundation compositions and exports.

Answer 1

Try this:

AVMutableVideoCompositionLayerInstruction *passThroughLayer = AVMutableVideoCompositionLayerInstruction videoCompositionLayerInstructionWithAssetTrack:videoTrack];
CGAffineTransform rotationTransform = CGAffineTransformMakeRotation(degreesToRadians(90.0));
CGAffineTransform rotateTranslate = CGAffineTransformTranslate(rotateTransform,320,0);
[passThroughLayer setTransform:rotateTranslate atTime:kCMTimeZero];

Essentially the idea is to create a rotation and translation matrix. You rotate it to the proper orientation and then translate it into the view. I did not see any way to specify a center point while I was glancing through the API.

Answer 2

Building on what was answered so far. I found a very good way of debugging and finding out what went wrong with your transforms. Using the ramp methods available, you are able to animate the transforms making it easier to see what your transform is doing.

Most of the time I found myself having transforms that appeared to do nothing until I realised that just using preferredTransform property of a video track alone may result in the video feed moving out of the render screen.

AVMutableVideoCompositionLayerInstruction *videoLayerInstruction = [AVMutableVideoCompositionLayerInstruction videoCompositionLayerInstructionWithAssetTrack:videoTrack];

[videoLayerInstruction setTransformRampFromStartTransform:CGAffineTransformIdentity
toEndTransform:videoTrack.preferredTransform 
timeRange:CMTimeRangeMake(projectClipStart, projectClipDuration)];

Eventually, I found that in some cases I needed to apply a translation to bring back the rotated video into the render screen.

CGAffineTransformConcat(videoTrack.preferredTransform, CGAffineTransformMakeTranslation(0, renderSize.height))

Note: Your translation values may be different.

Answer 3

AVAssetTrack *videoAssetTrack= [[videoAsset tracksWithMediaType:AVMediaTypeVideo] lastObject];

AVMutableCompositionTrack *videoCompositionTrack = [composition addMutableTrackWithMediaType:AVMediaTypeVideo preferredTrackID:kCMPersistentTrackID_Invalid];
[videoCompositionTrack insertTimeRange:CMTimeRangeMake(kCMTimeZero, videoAsset.duration) ofTrack:videoAssetTrack atTime:kCMTimeZero error:&error];

videoCompositionTrack.preferredTransform = videoAssetTrack.preferredTransform;

Answer 4

Also may be relevant, but there is a bug in the preferredTransform being set when using the front-facing camera .. see here for example, where the guys in the SDAVAssetExportSession project have coded a work-around:

https://github.com/rs/SDAVAssetExportSession/pull/70

Answer 5

I found solution in Flutter plugin project.

- (CGAffineTransform)fixTransform:(AVAssetTrack*)videoTrack {
  CGAffineTransform transform = videoTrack.preferredTransform;

  if (transform.tx == 0 && transform.ty == 0) {
      NSInteger rotationDegrees = (NSInteger)round(radiansToDegrees(atan2(transform.b, transform.a)));
      NSLog(@"TX and TY are 0. Rotation: %ld. Natural width,height: %f, %f", (long)rotationDegrees,
      videoTrack.naturalSize.width, videoTrack.naturalSize.height);
      if (rotationDegrees == 90) {
        NSLog(@"Setting transform tx");
        transform.tx = videoTrack.naturalSize.height;
        transform.ty = 0;
      } else if (rotationDegrees == 270) {
        NSLog(@"Setting transform ty");
        transform.tx = 0;
        transform.ty = videoTrack.naturalSize.width;
     }
  }
  return transform;
}

// set layerInstruction
[firstVideoLayerInstruction setTransform:[self fixTransform:firstVideoAssetTrack] atTime:kCMTimeZero];

Flutter VideoPlayerPlugin