How do you add to an AudioBufferList with an AVAssetReader?

Question

I have been working on reading in an audio asset using AVAssetReader so that I can later play back the audio with an AUGraph with an AudioUnit callback. I have the AUGraph and AudioUnit callback working but it reads files from disk and if the file is too big it would take up too much memory and crash the app. So I am instead reading the asset directly and only a limited size. I will then manage it as a double buffer and get the AUGraph what it needs when it needs it.

(Note: I would love know if I can use Audio Queue Services and still use an AUGraph with AudioUnit callback so memory is managed for me by the iOS frameworks.)

My problem is that I do not have a good understanding of arrays, structs and pointers in C. The part where I need help is taking the individual AudioBufferList which holds onto a single AudioBuffer and add that data to another AudioBufferList which holds onto all of the data to be used later. I believe I need to use memcpy but it is not clear how to use it or even initialize an AudioBufferList for my purposes. I am using MixerHost for reference which is the sample project from Apple which reads in the file from disk.

I have uploaded my work in progress if you would like to load it up in Xcode. I've figured out most of what I need to get this done and once I have the data being collected all in one place I should be good to go.

Sample Project: MyAssetReader.zip

In the header you can see I declare the bufferList as a pointer to the struct.

@interface MyAssetReader : NSObject {
    BOOL reading;
    signed long sampleTotal;
    Float64 totalDuration;

    AudioBufferList *bufferList; // How should this be handled?
}

Then I allocate bufferList this way, largely borrowing from MixerHost...

UInt32 channelCount = [asset.tracks count];

if (channelCount > 1) {
    NSLog(@"We have more than 1 channel!");
}

bufferList = (AudioBufferList *) malloc (
                                         sizeof (AudioBufferList) + sizeof (AudioBuffer) * (channelCount - 1)
                                         );

if (NULL == bufferList) {NSLog (@"*** malloc failure for allocating bufferList memory"); return;}

// initialize the mNumberBuffers member
bufferList->mNumberBuffers = channelCount;

// initialize the mBuffers member to 0
AudioBuffer emptyBuffer = {0};
size_t arrayIndex;
for (arrayIndex = 0; arrayIndex < channelCount; arrayIndex++) {
    // set up the AudioBuffer structs in the buffer list
    bufferList->mBuffers[arrayIndex] = emptyBuffer;
    bufferList->mBuffers[arrayIndex].mNumberChannels  = 1;
    // How should mData be initialized???
    bufferList->mBuffers[arrayIndex].mData            = malloc(sizeof(AudioUnitSampleType)); 
}

Finally I loop through the reads.

int frameCount = 0;

CMSampleBufferRef nextBuffer;
while (assetReader.status == AVAssetReaderStatusReading) {
    nextBuffer = [assetReaderOutput copyNextSampleBuffer];

    AudioBufferList  localBufferList;
    CMBlockBufferRef blockBuffer;    
    CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(nextBuffer, NULL, &localBufferList, sizeof(localBufferList), NULL, NULL, 
                                                            kCMSampleBufferFlag_AudioBufferList_Assure16ByteAlignment, &blockBuffer);

    // increase the number of total bites
    bufferList->mBuffers[0].mDataByteSize += localBufferList.mBuffers[0].mDataByteSize;

    // carefully copy the data into the buffer list
    memcpy(bufferList->mBuffers[0].mData + frameCount, localBufferList.mBuffers[0].mData, sizeof(AudioUnitSampleType));

    // get information about duration and position
    //CMSampleBufferGet
    CMItemCount sampleCount = CMSampleBufferGetNumSamples(nextBuffer);
    Float64 duration = CMTimeGetSeconds(CMSampleBufferGetDuration(nextBuffer));
    Float64 presTime = CMTimeGetSeconds(CMSampleBufferGetPresentationTimeStamp(nextBuffer));

    if (isnan(duration)) duration = 0.0;
    if (isnan(presTime)) presTime = 0.0;

    //NSLog(@"sampleCount: %ld", sampleCount);
    //NSLog(@"duration: %f", duration);
    //NSLog(@"presTime: %f", presTime);

    self.sampleTotal += sampleCount;
    self.totalDuration += duration;
    frameCount++;

    free(nextBuffer);
}

I am unsure about the what that I handle mDataByteSize and mData, especially with memcpy. Since mData is a void pointer this is an extra tricky area.

    memcpy(bufferList->mBuffers[0].mData + frameCount, localBufferList.mBuffers[0].mData, sizeof(AudioUnitSampleType));

In this line I think it should be copying the value from the data in localBufferList to the position in the bufferList plus the number of frames to position the pointer where it should write the data. I have a couple of ideas on what I need to change to get this to work.

Since a void pointer is just 1 and not the size of the pointer for an AudioUnitSampleType I may need to multiply it also by sizeof(AudioUnitSampleType) to get the memcpy into the right position
I may not be using malloc properly to prepare mData but since I am not sure how many frames there will be I am not sure what to do to initialize it

Currently when I run this app it ends this function with an invalid pointer for bufferList.

I appreciate your help with making me better understand how to manage an AudioBufferList.

I've been learning more about void pointers and how to use malloc and memcpy which seem to be a key part of understanding the AudioBufferList struct. I will need need to know a way to hold into the values into a dynamic array in C. I am reading features like this may have been added with C99. I am continuing to read up on it. — Brennan
I am thinking that I will accumulate the values from AudioBuffer to an array which is grown with ralloc but I may do this with blocks of 100 so realloc is not called with each iteration. Then I can create a dynamically sized array after the loop, use memcpy to populate that array with the accumulated values and then set mData on the buffer with this array. — Brennan

Brennan Brennan · Accepted Answer · 2011-04-21T07:41:21

I've come up with my own answer. I decided to use an NSMutableData object which allows me to appendBytes from the CMSampleBufferRef after calling CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer to get an AudioBufferList.

        [data appendBytes:localBufferList.mBuffers[0].mData length:localBufferList.mBuffers[0].mDataByteSize];

Once the read loop is done I have all of the data in my NSMutableData object. I then create and populate the AudioBufferList this way.

audioBufferList = (AudioBufferList *)malloc(sizeof(AudioBufferList));
if (NULL == audioBufferList) {
    NSLog (@"*** malloc failure for allocating audioBufferList memory");
    [data release];
    return;
}
audioBufferList->mNumberBuffers = 1;
audioBufferList->mBuffers[0].mNumberChannels = channelCount;
audioBufferList->mBuffers[0].mDataByteSize = [data length];
audioBufferList->mBuffers[0].mData = (AudioUnitSampleType *)malloc([data length]);
if (NULL == audioBufferList->mBuffers[0].mData) {
    NSLog (@"*** malloc failure for allocating mData memory");
    [data release];
    return;
}
memcpy(audioBufferList->mBuffers[0].mData, [data mutableBytes], [data length]);
[data release];

I'd appreciate a little code review on how I use malloc to create the struct and populate it. I am getting a EXC_BAD_ACCESS error sporadically but I cannot pinpoint where the error is just yet. Since I am using malloc on the struct I should not have to retain it anywhere. I do call "free" to release child elements within the struct and finally the struct itself everywhere that I use malloc.

How do you add to an AudioBufferList with an AVAssetReader?

1 Answers