I am working on a project in which I generate a video from UIImage, with code I found around here, and I am struggling for some days now in order to optimize it (for about 300 images, it takes like 5 minutes on the simulator, and simply crashes on device because of the memory).
I'll begin with the working code I have today (I work with arc):
-(void) writeImageAsMovie:(NSArray *)array toPath:(NSString*)path size:(CGSize)size duration:(int)duration
{
NSError *error = nil;
AVAssetWriter *videoWriter = [[AVAssetWriter alloc] initWithURL:[NSURL fileURLWithPath:path] fileType:AVFileTypeQuickTimeMovie error:&error];
NSParameterAssert(videoWriter);
NSDictionary *videoSettings = [NSDictionary dictionaryWithObjectsAndKeys:
AVVideoCodecH264, AVVideoCodecKey,
[NSNumber numberWithInt:size.width], AVVideoWidthKey,
[NSNumber numberWithInt:size.height], AVVideoHeightKey,
nil];
AVAssetWriterInput* writerInput = [AVAssetWriterInput
assetWriterInputWithMediaType:AVMediaTypeVideo
outputSettings:videoSettings];
AVAssetWriterInputPixelBufferAdaptor *adaptor = [AVAssetWriterInputPixelBufferAdaptor assetWriterInputPixelBufferAdaptorWithAssetWriterInput:writerInput sourcePixelBufferAttributes:nil];
NSParameterAssert(writerInput);
NSParameterAssert([videoWriter canAddInput:writerInput]);
[videoWriter addInput:writerInput];
//Start a session:
[videoWriter startWriting];
[videoWriter startSessionAtSourceTime:kCMTimeZero];
CVPixelBufferRef buffer = NULL;
buffer = [self newPixelBufferFromCGImage:[[self.frames objectAtIndex:0] CGImage]];
CVPixelBufferPoolCreatePixelBuffer (NULL, adaptor.pixelBufferPool, &buffer);
[adaptor appendPixelBuffer:buffer withPresentationTime:kCMTimeZero];
dispatch_queue_t mediaInputQueue = dispatch_queue_create("mediaInputQueue", NULL);
int frameNumber = [self.frames count];
[writerInput requestMediaDataWhenReadyOnQueue:mediaInputQueue usingBlock:^{
NSLog(@"Entering block with frames: %i", [self.frames count]);
if(!self.frames || [self.frames count] == 0)
{
return;
}
int i = 1;
while (1)
{
if (i == frameNumber)
{
break;
}
if ([writerInput isReadyForMoreMediaData])
{
freeMemory();
NSLog(@"inside for loop %d (%i)",i, [self.frames count]);
UIImage *image = [self.frames objectAtIndex:i];
CGImageRef imageRef = [image CGImage];
CVPixelBufferRef sampleBuffer = [self newPixelBufferFromCGImage:imageRef];
CMTime frameTime = CMTimeMake(1, TIME_STEP);
CMTime lastTime=CMTimeMake(i, TIME_STEP);
CMTime presentTime=CMTimeAdd(lastTime, frameTime);
if (sampleBuffer)
{
[adaptor appendPixelBuffer:sampleBuffer withPresentationTime:presentTime];
i++;
CVPixelBufferRelease(sampleBuffer);
}
else
{
break;
}
}
}
[writerInput markAsFinished];
[videoWriter finishWriting];
self.frames = nil;
CVPixelBufferPoolRelease(adaptor.pixelBufferPool);
}];
}
And now the function to get the pixel buffer, with which I'm struggling:
- (CVPixelBufferRef) newPixelBufferFromCGImage: (CGImageRef) image
{
CVPixelBufferRef pxbuffer = NULL;
int width = CGImageGetWidth(image)*2;
int height = CGImageGetHeight(image)*2;
NSMutableDictionary *attributes = [NSMutableDictionary dictionaryWithObjectsAndKeys:[NSNumber kCVPixelFormatType_32ARGB], kCVPixelBufferPixelFormatTypeKey, [NSNumber numberWithInt:width], kCVPixelBufferWidthKey, [NSNumber numberWithInt:height], kCVPixelBufferHeightKey, nil];
CVPixelBufferPoolRef pixelBufferPool;
CVReturn theError = CVPixelBufferPoolCreate(kCFAllocatorDefault, NULL, (__bridge CFDictionaryRef) attributes, &pixelBufferPool);
NSParameterAssert(theError == kCVReturnSuccess);
CVReturn status = CVPixelBufferPoolCreatePixelBuffer(NULL, pixelBufferPool, &pxbuffer);
NSParameterAssert(status == kCVReturnSuccess && pxbuffer != NULL);
CVPixelBufferLockBaseAddress(pxbuffer, 0);
void *pxdata = CVPixelBufferGetBaseAddress(pxbuffer);
NSParameterAssert(pxdata != NULL);
CGColorSpaceRef rgbColorSpace = CGColorSpaceCreateDeviceRGB();
CGContextRef context = CGBitmapContextCreate(pxdata, width,
height, 8, width*4, rgbColorSpace,
kCGImageAlphaNoneSkipFirst);
NSParameterAssert(context);
CGContextDrawImage(context, CGRectMake(0, 0, width,
height), image);
CGColorSpaceRelease(rgbColorSpace);
CGContextRelease(context);
CVPixelBufferUnlockBaseAddress(pxbuffer, 0);
return pxbuffer;
}
First strange thing: as you can see in this function, I have to multiply the width and height by 2, otherwise, the result video is all messed up, and I can't understand why (I can post screenshots if it helps; the pixels seem to come from my image, but the width is not correct, and there is a large black square on the half bottom of the video).
Another problem is that it takes a really big amount of memory; I think that the pixel buffer could not be well deallocated, but I don't see why.
Finally, it is very slow, but I have two ideas to improve it, which I fail to use.
The first is to avoid using UIImage to create my pixel buffers, since I generate the UIImage myself with (uint8_t *) data. I tried to use 'CVPixelBufferCreateWithBytes', but it wouldn't work. Here is how I tried it:
OSType pixFmt = CVPixelBufferGetPixelFormatType(pxbuffer); CVPixelBufferCreateWithBytes(kCFAllocatorDefault, width, height, pixFmt, self.composition.srcImage.resultImageData, width*2, NULL, NULL, (__bridge CFDictionaryRef) attributes, &pxbuffer);
(The arguments are the same as for the above functions; my image data are encoded in 16 bits per pixel, and I couldn't find a good OSType argument to give to the function.) If someone know how to use it (maybe it's not possible with 16 bits/pixel data?), it would help me to avoid a really useless conversion.
- The second thing is that I would like to avoid kCVPixelFormatType_32ARGB for my video. I guess it would be quicker to use something with fewer bits/pixel, but when I try it (I have tried all the kCVPixelFormatType_16XXXXX formats, with a context created with 5 bits/component and kCGImageAlphaNoneSkipFirst), either it crashes, either the resulting video contains nothing (with kCVPixelFormatType_16BE555).
I know I ask a lot in only one post, but I am kind of lost in this code, I have tried so many combinations and none of them worked...
