How to Skip frames while decoding H264 stream?

Question

I'm using FFMPEG to decode H264 (or H265) RTSP Stream.

My system have 2 software: Server and Client

Server: Read frames from RTSP stream --> Forward frames to Client    
Client: Receive frames from Server --> Decode --> Render

I have implemented and it worked ok, but there is a case make my system work not good. That is when internet from Server - Client is slow, frames can not transfer real-time to Client.

In present, I deal with this issue by Skip some frames (not send to Client) when the Queue is reached limit of count. The following is my summary code

//At Server Software (include 2 threads A and B)
//Thread A: Read AVPacket and forward to Client
while(true)
{
    AVPacket packet;
    av_init_packet(&packet);
    packet.size = 0;
    packet.data = NULL;
    int ret = AVERROR(EAGAIN);
    while (AVERROR(EAGAIN) == ret)
        ret = av_read_frame(pFormatCtx, &packet);
    if(packet.size > 0)
    {
        if(mySendQueue.count < 120) //limit 120 packet in queue
            mySendQueue.Enqueue(packet); ////Thread B will read from this queue, to send packets to Client via TCP socket
        else
            ;//SkipThisFrame ***: No send
    }
}
//Thread B: Send To Client via TCP Socket
While(true)
{
    AVPacket packet;
    if(mySendQueue.Dequeue(packet))
    {
        SendPacketToClient(packet);
    }
}

//At Server Software : Receive AVPacket from Server --> Decode --> Render
While(true)
{
    AVPacket packet;
    AVFrame frame;
    ReadPacketFromServer(packet);
    if (av_decode_asyn(pCodecCtx, &frame, &frameFinished, &packet) == RS_OK)
    {
        if (frameFinished)
        {
            RenderFrame(frame);
        }
    }           
}
UINT32 __clrcall av_decode_asyn(AVCodecContext *pCodecCtx, AVFrame *frame, int *frameFinished, AVPacket *packet)
{
    int ret = -1;
    *frameFinished = 0;
    if (packet) 
    {
        ret = avcodec_send_packet(pCodecCtx, packet);
        // In particular, we don't expect AVERROR(EAGAIN), because we read all
        // decoded frames with avcodec_receive_frame() until done.
        if (ret < 0 && ret != AVERROR_EOF)
            return RS_NOT_OK;
    }

    ret = avcodec_receive_frame(pCodecCtx, frame);
    if (ret < 0 && ret != AVERROR(EAGAIN))
    {
        return RS_NOT_OK;
    }
    if (ret >= 0)
        *frameFinished = 1;

    return RS_OK;
}

My question is focus in line of code SkipThisFrame ***, this algorithm skip frame continuously, so it maybe make the decoder on Client occur unexpectedly error or Crash?

And when skip frame like that, make Client Render frames is not normally?

And someone call show me the proper algorithm to skip frames in my case?

Thank you very much!

Shouldn't that logic be Server: Store "incoming" frames from RTSP stream --> Forward "existing" frames to Client? — VC.One
@VC.One: Thank you! I understand that you said, but when internet is slow, I have to skip frames for real-time render on Client. I consider that which algorithm of skip frame is best — TTGroup
As other already have stated, you cannot skip decoding frames in a h264 stream since a P-frame (predicted) just contains the differences towards the last reconstructed one. However, what you can do is to skip showing frames on the display in question and limit the perceived fps. — Fredrik Pihl

jiandingzhe jiandingzhe · Accepted Answer · 2018-09-13T09:27:33

I have a brief read on doc of AVPacket, it says:

For video, it should typically contain one compressed frame.

Theoretically you cannot skip frames for a compressed video stream, as most frames do not contain complete information about that frame's image, but only contain changes compared with some previous frames. So if you skip a frame, it is probable that many trailing decoded frames won't contain correct result (until next key frame flushes whole image).

How to Skip frames while decoding H264 stream?

2 Answers