Audio/video out of sync after switch camera

Question

I'm try create application where I can record video from different cameras on device during recording. For example. User press button "start record" from front camera, after 5 second recording user press button "Switch Camera" and application change video source from front to back camera and recording continue. For camera swithcing I use next code:

NSError *error;
AVCaptureDeviceInput *newVideoInput;
AVCaptureDevicePosition currentCameraPosition = [[videoInput device] position];

if (currentCameraPosition == AVCaptureDevicePositionBack)
{
    currentCameraPosition = AVCaptureDevicePositionFront;
}
else
{
    currentCameraPosition = AVCaptureDevicePositionBack;
}

AVCaptureDevice *backFacingCamera = nil;
NSArray *devices = [AVCaptureDevice devicesWithMediaType:AVMediaTypeVideo];
for (AVCaptureDevice *device in devices) 
{
    if ([device position] == currentCameraPosition)
    {
        backFacingCamera = device;
    }
}
newVideoInput = [[AVCaptureDeviceInput alloc] initWithDevice:backFacingCamera error:&error];

if (newVideoInput != nil)
{
    [_captureSession beginConfiguration];

    [_captureSession removeInput:videoInput];
    if ([_captureSession canAddInput:newVideoInput])
    {
        [_captureSession addInput:newVideoInput];
        videoInput = newVideoInput;
    }
    else
    {
        [_captureSession addInput:videoInput];
    }
    //captureSession.sessionPreset = oriPreset;
    [_captureSession commitConfiguration];
}

_inputCamera = backFacingCamera;

After this appication change video sourse and continue writing, but... audio/video out of sync... Can anybody hehp me with this problem?

Thank you.

Answer 1

you need to stop recording, switch and start it again. the camera switch isnt instant AFAIK

cant you record into n files and later stich them together?

try using individual AVMutableComposition tracks and then setting a mutablecomposition for audio and one for video. (see Merging two m4v Movie Files Using AVMutableComposition - Videos Will Not Merge)

Answer 2

I had the same problem, tried a lot if things and came up with an easy and convenient solution which works without any glitch.

The problem is, as @daij-djan pointed out, that switching a session input takes a little time and adds a few black frames to the output and then continue sending frames as if it has never stop. As far as I know it is impossible to know how much frames are impacted. The session timecode is not impacted by this delay so we can't use it to ignore some video frames.

Instead of having one session with multiple camera inputs, you can have one session per camera input (+ one for audio if you need it) and one video output. Then you only need to switch this output between the sessions.

As a result you will have no desynchronization and no ugly hack to do. The memory impact is very limited in my tests and I didn't noticed any performance impact. My theory is that a session isn't active until it has an output attached.

Switching outputs can be done like this:

fromSession.beginConfiguration()
fromSession.removeOutput(videoOutput)
fromSession.commitConfiguration()

toSession.beginConfiguration()
if toSession.canAddOutput(videoOutput) { toSession.addOutput(videoOutput) }
toSession.commitConfiguration()

Answer 3

when switching camera, you need to pause the buffer writing in the delegate method:

- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection

you can checkout the open source project PBJVision