7
votes

I am writing program for streaming live audio and video from webcamera to rtmp-server. I work in MacOS X 10.8, so I use AVFoundation framework for obtaining audio and video frames from input devices. This frames come into delegate:

-(void) captureOutput:(AVCaptureOutput*)captureOutput didOutputSampleBuffer: (CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection*)connection ,

where sampleBuffer contains audio or video data.

When I recieve audio data in the sampleBuffer, I'm trying to convert this data into AVFrame and encode AVFramewith libavcodec:

    aframe = avcodec_alloc_frame();  //AVFrame *aframe;
    int got_packet, ret;
    CMItemCount numSamples = CMSampleBufferGetNumSamples(sampleBuffer); //CMSampleBufferRef

    NSUInteger channelIndex = 0;

    CMBlockBufferRef audioBlockBuffer = CMSampleBufferGetDataBuffer(sampleBuffer);

    size_t audioBlockBufferOffset = (channelIndex * numSamples * sizeof(SInt16));

    size_t lengthAtOffset = 0;

    size_t totalLength = 0;

    SInt16 *samples = NULL;

    CMBlockBufferGetDataPointer(audioBlockBuffer, audioBlockBufferOffset, &lengthAtOffset, &totalLength, (char **)(&samples));

            const AudioStreamBasicDescription *audioDescription = CMAudioFormatDescriptionGetStreamBasicDescription(CMSampleBufferGetFormatDescription(sampleBuffer));

    aframe->nb_samples =(int) numSamples;

    aframe->channels=audioDescription->mChannelsPerFrame;

    aframe->sample_rate=(int)audioDescription->mSampleRate;

     //my webCamera configured to produce 16bit 16kHz LPCM mono, so sample format hardcoded here, and seems to be correct
    avcodec_fill_audio_frame(aframe, aframe->channels, AV_SAMPLE_FMT_S16,

                             (uint8_t *)samples,

                              aframe->nb_samples *

                             av_get_bytes_per_sample(AV_SAMPLE_FMT_S16) *

                             aframe->channels, 0);  
    //encoding audio
    ret = avcodec_encode_audio2(c, &pkt, aframe, &got_packet);
    if (ret < 0) {
        fprintf(stderr, "Error encoding audio frame: %s\n", av_err2str(ret));
        exit(1);
    }

The problem is that when I get so formed frames, I can hear the wanted sound, but it is slowing down and discontinuous (as if after each data frame comes the same frame of silence). It seems that something is wrong in the transformation from CMSampleBuffer to AVFrame , because the preview from the microphone created with AVFoundation from the same sample buffers played normally.

I would be grateful for your help.

UPD: Creating and initializing the AVCodceContext structure

    audio_codec= avcodec_find_encoder(AV_CODEC_ID_AAC);
    if (!(audio_codec)) {
        fprintf(stderr, "Could not find encoder for '%s'\n",
                avcodec_get_name(AV_CODEC_ID_AAC));
        exit(1);
    }
    audio_st = avformat_new_stream(oc, audio_codec);  //AVFormatContext *oc;
    if (!audio_st) {
        fprintf(stderr, "Could not allocate stream\n");
        exit(1);
    }

      audio_st->id=1;
      audio_st->codec->sample_fmt= AV_SAMPLE_FMT_S16;
      audio_st->codec->bit_rate = 64000;
      audio_st->codec->sample_rate= 16000;
      audio_st->codec->channels=1;
      audio_st->codec->codec_type= AVMEDIA_TYPE_AUDIO;

Describe how you create and initialize the AVCodceContext structure - pogorskiy
For test purposes, can clearly true initialize AVFrame structure in accordance with a predetermined format, for example sinusoidal signal, and validate the output. - pogorskiy
I created audio frame as presented in ffmpeg example (ffmpeg/doc/samples/muxing.c). With sinusoidal signal, output sounds correct. But when I take audio-samples from microphone instead of sinusoidal samles, problem arises. Also I have found, that if I use 'AV_CODEC_ID_PCM_ALAW' instead of 'AV_CODEC_ID_AAC', output signal sounds slow, but without discontinuities. - Aleksei2414904
Slow signal means that the output samplerate does not match the input. The discontinuity for the AAC codec comes from the fact that the AAC accepts input only planar float data (AV_SAMPLE_FMT_FLTP) (each channel is separated from the others) - pogorskiy
I tried to initialize the codec with AV_SAMPLE_FMT_FLTP, but received error [libfaac @ 0x104121c00] Specified sample format fltp is invalid or not supported Could not open audio codec: Invalid argument during performing avcodec_open2. I also checked for all other sample formats, and found that the codec successfully initialized only if I use the AV_SAMPLE_FMT_P16 or AV_SAMPLE_FMT _S16P. But in both cases the problem with discontinuity remains. - Aleksei2414904