LIVE555 how to use h264 framer class to get nal units for ffmpeg

Question

I'm trying to create a small app which will save frames from inoming h264 stream. I took a testRTSP programm as example and made several changes in DummySink::afterGettingFrame function to decode frames with the help of ffmpeg library. As I understand from frameSize, my first two frames are SPS units, so I concatenate them with my third frame and then send new big frame to ffmpeg decoder. But that doesnt work. ffmpeg tells me that my first frame is too big for SPS and then it tells me that there is no frame... I dont know what I need to change here.

void DummySink::afterGettingFrame(unsigned frameSize, unsigned numTruncatedBytes,
struct timeval presentationTime, unsigned /*durationInMicroseconds*/)
{
u_int8_t start_code[4] = { 0x00, 0x00, 0x00, 0x01 };
int stCodeLen = 4;

if (frameSize == 50)
{
    //add start code
    memcpy(bufferWithStartCode, start_code, stCodeLen);
    shiftPtr += stCodeLen;
    memcpy(bufferWithStartCode + shiftPtr, fReceiveBuffer, frameSize);
    shiftPtr += frameSize;
}
else if (frameSize == 4)
{
    memcpy(bufferWithStartCode + shiftPtr, fReceiveBuffer, frameSize);
    shiftPtr += frameSize;
}
else
{
    if (shiftPtr == 0)
    {
        memcpy(bufferWithStartCode, start_code, stCodeLen);
        shiftPtr += stCodeLen;
    }
    memcpy(bufferWithStartCode + shiftPtr, fReceiveBuffer, frameSize);
    avpkt.size = frameSize + shiftPtr;
    avpkt.data = bufferWithStartCode;
    shiftPtr = 0;
    if (!avcodec_send_packet(cContext, &avpkt))
    {
        envir() << "error sending to decoder";

    }
    if (!avcodec_receive_frame(cContext, picture))
    {
        envir() << "error rx from decoder";
    }
    if (picture)
    {
        FILE *f;
        char buffer[32]; // The filename buffer.
        snprintf(buffer, sizeof(char) * 32, "file%i.txt", frame_num);
        f = fopen(buffer, "w");
        fprintf(f, "P5\n%d %d\n%d\n", fSubsession.videoWidth(), fSubsession.videoHeight(), 255);
        for (int i = 0;i < fSubsession.videoHeight();i++)
            fwrite(picture->data[0] + i * (picture->linesize[0]), 1, fSubsession.videoWidth(), f);
        fclose(f);
    }
}

envir() << frameSize << "\n";


frame_num++;

// Then continue, to request the next frame of data:
continuePlaying();

Aleksey Aleksey · Accepted Answer · 2016-08-02T08:14:25

I've found solution for my problem.

There is function void DummySink::afterGettingFrame(...) In "testRTSP" example of live555. All I was need to do is to assembly my frame everytime when function got frame:

[start_code sps pps start_code frame_data]

frame_data is fReceiveBuffer at this point. start_code is char array [0,0,0,1].

And push new data to ffmpeg decoder:

m_packet.size = frameBufSize + frameSize; // size of assembled frame
m_packet.data = frameBuf; // assembled frame

if (avcodec_send_packet(m_decoderContext, &m_packet) != 0)
{
    envir() << "error in sending packet to decoder" << "\n";
}
if (avcodec_receive_frame(m_decoderContext, pFrame) == 0)

No additional settings for decoderContext! Just init everything like in tutorials (http://dranger.com/ffmpeg/ - this is up-to-date tutorials for ffmpeg c++ library) and you good to go. I used memcpy to unite data in one big array. memcpy(frameBuf, startCode, 4); frameBufSize += 4;

for (int i = 0; i < numSPropRecords; i++)
{
    memcpy(frameBuf + frameBufSize, sPropRecords[i].sPropBytes, sPropRecords[i].sPropLength);
    frameBufSize += sPropRecords[i].sPropLength;
}

memcpy(frameBuf + frameBufSize, startCode, 4);
frameBufSize += 4;
memcpy(frameBuf + frameBufSize, fReceiveBuffer, frameSize);

m_packet.size = frameBufSize + frameSize;
m_packet.data = frameBuf;

You can get sps and pps data from subsession (check "openRTSP" for detailed example or "H264orH264FileSink.h")

spsppsunits = subsession.fmtp_spropparametersets();
sPropRecords = parseSPropParameterSets(spsppsunits, numSPropRecords);

UPDATE 1:

After some time I found errors in my approach to ffmpeg. Sometimes ffmpeg decoder wont work with H264 stream if you wont supply extradata information with sps and pps units. So,

m_decoderContext->extradata =  (uint8_t*)av_malloc(100 + AV_INPUT_BUFFER_PADDING_SIZE);
int extraDataSize = 0;
for (int i = 0; i < numSPropRecords; i++)
{
    memcpy(m_decoderContext->extradata + extraDataSize, startCode, 4);
    extraDataSize += 4;
    memcpy(m_decoderContext->extradata + extraDataSize, sPropRecords[i].sPropBytes, sPropRecords[i].sPropLength);
    extraDataSize += sPropRecords[i].sPropLength;
}
m_decoderContext->extradata_size = extraDataSize;

After that you dont need to supply frame with sps and pps data every time, only start code is needed.

LIVE555 how to use h264 framer class to get nal units for ffmpeg

2 Answers