MediaCodec decode AAC audio chunks from RTSP and play

Question

I'm receiving rtp packets containing aac audio chunks encoded by libvo_aacenc (44100hz 128kbps 2ch) from a FFServer instance. I'm trying to decode them with MediaCodec one by one in Android and playback as soon as the chunk is decoded.

Client.java

Player player = new Player();
//RTSP listener
@Override
public void onRTSPPacketReceived(RTPpacket packet) {
    byte [] aac_chunk = packet.getpayload();
    player.playAAC(aac_chunk);
}

Player.java

private MediaCodec decoder;
private AudioTrack audioTrack;
private MediaExtractor extractor;

public Player(){
    extractor = new MediaExtractor();
    audioTrack = new AudioTrack(AudioManager.STREAM_MUSIC,
            44100, AudioFormat.CHANNEL_OUT_STEREO,
            AudioFormat.ENCODING_PCM_16BIT,
            44100,
            AudioTrack.MODE_STREAM);

    MediaFormat format = new MediaFormat();
    format.setString(MediaFormat.KEY_MIME, "audio/mp4a-latm");
    format.setInteger(MediaFormat.KEY_BIT_RATE, 128 * 1024);
    format.setInteger(MediaFormat.KEY_CHANNEL_COUNT, 2);
    format.setInteger(MediaFormat.KEY_SAMPLE_RATE, 44100);
    format.setInteger(MediaFormat.KEY_AAC_PROFILE, MediaCodecInfo.CodecProfileLevel.AACObjectHE);

    try{
        decoder = MediaCodec.createDecoderByType("audio/mp4a-latm");
        decoder.configure(format, null, null, 0);
    } catch (IOException e) {
        e.printStackTrace();
    }

    decoder.start();
    audioTrack.play();
}

//Decode and play one aac_chunk
public void playAAC(byte [] data){

    MediaCodec.BufferInfo info = new MediaCodec.BufferInfo();
    ByteBuffer[] inputBuffers = decoder.getInputBuffers();
    ByteBuffer[] outputBuffers = decoder.getOutputBuffers();

    int inIndex = decoder.dequeueInputBuffer(-1);
    if (inIndex >= 0) {

        ByteBuffer buffer = inputBuffers[inIndex];
        buffer.put(data, 0, data.length);

        int sampleSize = extractor.readSampleData(buffer, 0);

        if (sampleSize < 0) {
            decoder.queueInputBuffer(inIndex, 0, 0, 0, MediaCodec.BUFFER_FLAG_END_OF_STREAM);
        } else {
            long presentationTimeUs = extractor.getSampleTime();
            decoder.queueInputBuffer(inIndex, 0, sampleSize, presentationTimeUs, 0);
        }
    }

    int outIndex = decoder.dequeueOutputBuffer(info, TIMEOUT);
    while(outIndex >= 0){
        ByteBuffer outBuffer = outputBuffers[outIndex];
        byte[] decoded_chunk = new byte[info.size];
        outBuffer.get(decoded_chunk); // Read the buffer all at once
        outBuffer.clear(); 

        //!! Decoded decoded_chunk.length = 0 !! 
        System.out.println("DECODED CHUNK SIZE: "+decoded_chunk.length);

        //Instant play of the decoded chunk
        audioTrack.write(decoded_chunk, info.offset, info.offset + info.size); 

        decoder.releaseOutputBuffer(outIndex, false);
        break;
    }
    decoder.flush();
}

On start, MediaCodec is correctly initiated.

MediaCodec: (0xa5040280) start
MediaCodec: (0xa5040280) input buffers allocated
MediaCodec: (0xa5040280) numBuffers (4)
MediaCodec: (0xa5040280) output buffers allocated
MediaCodec: (0xa5040280) numBuffers (4)

The problem

I'm actually hearing no sound. MediaCodec is working but looks like It's not decoding anything into his Output buffers, since decoded_chunk.length = 0 and outBuffer.limit() = 0 .

Questions

Should I async fill MediaCodec input buffers? Unfortunately I didn't find anything in the examples I found about this problem: instant decode and playback.

I've follow these examples:

Decode and playback AAC file extracting media information. (link)
Same but different way to implement MediaCodec, steps defined (link)

Nicolò Nicolò · Accepted Answer · 2018-02-04T23:19:32

I've solved this using MediaCodec in async mode and MediaCodec.Callback as described in the official docs here which is available only for Android minSdkVersion 21.

Basically I've used a queue for every RTP audio chunk I receive and then I'm notified every time MediaCodec buffers state change. It's actually easier to handle the decoder flow.

decoder.setCallback(new MediaCodec.Callback() {
       @Override
       public void onInputBufferAvailable(@NonNull MediaCodec mediaCodec, int i) {
             //One InputBuffer is available to decode 
             while (true) {
                  if(queue.size() > 0) {
                      byte[] data = queue.removeFirst();
                      MediaCodec.BufferInfo info = new MediaCodec.BufferInfo();
                      ByteBuffer buffer = mediaCodec.getInputBuffer(i);
                      buffer.put(data, 0, data.length);
                      mediaCodec.queueInputBuffer(i, 0, data.length, 0, 0);
                      break;
                   }
              }
        }

        @Override
        public void onOutputBufferAvailable(@NonNull MediaCodec mediaCodec, int i, @NonNull MediaCodec.BufferInfo info) {
              //DECODING PACKET ENDED
              ByteBuffer outBuffer = mediaCodec.getOutputBuffer(i);
              byte[] chunk = new byte[info.size];
              outBuffer.get(chunk); // Read the buffer all at once
              outBuffer.clear(); 
              audioTrack.write(chunk, info.offset, info.offset + info.size); // AudioTrack write data
              mediaCodec.releaseOutputBuffer(i, false);
        }

        @Override
        public void onError(@NonNull MediaCodec mediaCodec, @NonNull MediaCodec.CodecException e) {}

        @Override
        public void onOutputFormatChanged(@NonNull MediaCodec mediaCodec, @NonNull MediaFormat mediaFormat) {}
});

MediaCodec decode AAC audio chunks from RTSP and play

1 Answers