Does the Remote I/O audio unit set the number of channels in the buffer?

Question

I'm using kxmovie (it's a ffmpeg-based video player) as a base for an app and I'm trying to figure out how the RemoteI/O unit works on iOS when the only thing connected to a device is headphones and we're playing a track with more than 2 channels (say a surround 6 track channel). It seems like it is going with the output channel setting and the buffer only has 2 channels. Is this because of Core Audio's pull structure? And if so, what's happening to the other channels in the track? Are they being downmixed or simply ignored?

The code for the render callback connected to the remoteio unit is here:

- (BOOL) renderFrames: (UInt32) numFrames
               ioData: (AudioBufferList *) ioData
{
    NSLog(@"Number of channels in buffer: %lu",ioData->mNumberBuffers);

    for (int iBuffer=0; iBuffer < ioData->mNumberBuffers; ++iBuffer) {
        memset(ioData->mBuffers[iBuffer].mData, 0, ioData->mBuffers[iBuffer].mDataByteSize);
    }


    if (_playing && _outputBlock ) {

        // Collect data to render from the callbacks
        _outputBlock(_outData, numFrames, _numOutputChannels);

        // Put the rendered data into the output buffer
        if (_numBytesPerSample == 4) // then we've already got floats
        {
            float zero = 0.0;

            for (int iBuffer=0; iBuffer < ioData->mNumberBuffers; ++iBuffer) {

                int thisNumChannels = ioData->mBuffers[iBuffer].mNumberChannels;

                for (int iChannel = 0; iChannel < thisNumChannels; ++iChannel) {
                    vDSP_vsadd(_outData+iChannel, _numOutputChannels, &zero, (float *)ioData->mBuffers[iBuffer].mData, thisNumChannels, numFrames);
                }
            }
        }
        else if (_numBytesPerSample == 2) // then we need to convert SInt16 -> Float (and also scale)
        {
            float scale = (float)INT16_MAX;
            vDSP_vsmul(_outData, 1, &scale, _outData, 1, numFrames*_numOutputChannels);

            for (int iBuffer=0; iBuffer < ioData->mNumberBuffers; ++iBuffer) {

                int thisNumChannels = ioData->mBuffers[iBuffer].mNumberChannels;

                for (int iChannel = 0; iChannel < thisNumChannels; ++iChannel) {
                    vDSP_vfix16(_outData+iChannel, _numOutputChannels, (SInt16 *)ioData->mBuffers[iBuffer].mData+iChannel, thisNumChannels, numFrames);
                }
            }

        }        
    }

    return noErr;
}

Thanks!

edit: Here's the code for the ASBD (_ouputFormat). It's getting its values straight from the remoteio. You can also check the whole method file here.

if (checkError(AudioUnitGetProperty(_audioUnit,
                                    kAudioUnitProperty_StreamFormat,
                                    kAudioUnitScope_Input,
                                    0,
                                    &_outputFormat,
                                    &size),
               "Couldn't get the hardware output stream format"))
    return NO;


_outputFormat.mSampleRate = _samplingRate;
if (checkError(AudioUnitSetProperty(_audioUnit,
                                    kAudioUnitProperty_StreamFormat,
                                    kAudioUnitScope_Input,
                                    0,
                                    &_outputFormat,
                                    size),
               "Couldn't set the hardware output stream format")) {

    // just warning
}

_numBytesPerSample = _outputFormat.mBitsPerChannel / 8;
_numOutputChannels = _outputFormat.mChannelsPerFrame;

NSLog(@"Current output bytes per sample: %ld", _numBytesPerSample);
NSLog(@"Current output num channels: %ld", _numOutputChannels);

// Slap a render callback on the unit
AURenderCallbackStruct callbackStruct;
callbackStruct.inputProc = renderCallback;
callbackStruct.inputProcRefCon = (__bridge void *)(self);

if (checkError(AudioUnitSetProperty(_audioUnit,
                                    kAudioUnitProperty_SetRenderCallback,
                                    kAudioUnitScope_Input,
                                    0,
                                    &callbackStruct,
                                    sizeof(callbackStruct)),
               "Couldn't set the render callback on the audio unit"))
    return NO;

If memory serves, you provide are responsible for supplying the AudiobufferList and the non-interleaved buffers it points to. You can fill them if you like. The RemoteIO unit is only likely to look at the first two. — marko
@user523234, I added the ADBD section to the original post and also a link to the full method file. — awfulcode
@marko I'll try adding white noise to the surround channels in a surround file to see if that's what's happening. I did try to log how many channels were being buffered and it only shows two. — awfulcode

awfulcode awfulcode · Accepted Answer · 2013-11-25T21:33:53

I finally found the piece of code that's making it remix channels to stereo. It sets a property in KxAudioManager using the ASBD of the RIO. And then, in KxMovieDecoder.m, it sets ffmpeg options using that same variable. Here's the code:

id<KxAudioManager> audioManager = [KxAudioManager audioManager];
swrContext = swr_alloc_set_opts(NULL,
                                av_get_default_channel_layout(audioManager.numOutputChannels),
                                AV_SAMPLE_FMT_S16,
                                audioManager.samplingRate,
                                av_get_default_channel_layout(codecCtx->channels),
                                codecCtx->sample_fmt,
                                codecCtx->sample_rate,
                                0,
                                NULL);

Now it's off to do some reading on how ffmpeg is doing the decoding. Fun times.

Does the Remote I/O audio unit set the number of channels in the buffer?

1 Answers