C++ ffmpeg video missing frames and won't play in Quicktime

Question

I wrote some C++ code that uses ffmpeg to encode a video. I'm having two strange issues:

The final video is always missing 1 frame. That is, if I have it encode 10 frames the final video only has 9 (at least that's what ffprobe -show_frames -pretty $VIDEO | grep -F '[FRAME]' | wc -l tells me.
The final video plays fine in some players (mpv and vlc) but not in Quicktime. Quicktime just shows a completely black screen.

My code is roughly this (modified a bit to remove types that are unique to our code base):

First, I open the video file, write the headers and initialize things:

template <class PtrT>
using UniquePtrWithDeleteFunction = std::unique_ptr<PtrT, std::function<void (PtrT*)>>;


std::unique_ptr<FfmpegEncodingFrameSink> FfmpegEncodingFrameSink::Create(
    const std::string& dest_url) {
  AVFormatContext* tmp_format_ctxt;
  auto alloc_format_res = avformat_alloc_output_context2(&tmp_format_ctxt, nullptr, "mp4", dest_url.c_str());
  if (alloc_format_res < 0) {
    throw FfmpegException("Error opening output file.");
  }
  auto format_ctxt = UniquePtrWithDeleteFunction<AVFormatContext>(
      tmp_format_ctxt, CloseAvFormatContext);

  AVStream* out_stream_video = avformat_new_stream(format_ctxt.get(), nullptr);
  if (out_stream_video == nullptr) {
    throw FfmpegException("Could not create outputstream");
  }

  auto codec_context = GetCodecContext(options);
  out_stream_video->time_base = codec_context->time_base;

  auto ret = avcodec_parameters_from_context(out_stream_video->codecpar, codec_context.get());
  if (ret < 0) {
    throw FfmpegException("Failed to copy encoder parameters to outputstream");
  }

  if (!(format_ctxt->oformat->flags & AVFMT_NOFILE)) {
    ret = avio_open(&format_ctxt->pb, dest_url.c_str(), AVIO_FLAG_WRITE);
    if (ret < 0) {
      throw VideoDecodeException("Could not open output file: " + dest_url);
    }
  }

  ret = avformat_init_output(format_ctxt.get(), nullptr);
  if (ret < 0) {
    throw FfmpegException("Unable to initialize the codec.");
  }

  ret = avformat_write_header(format_ctxt.get(), nullptr);
  if (ret < 0) {
    throw FfmpegException("Error occurred writing format header");
  }

  return std::unique_ptr<FfmpegEncodingFrameSink>(
      new FfmpegEncodingFrameSink(std::move(format_ctxt), std::move(codec_context)));
}

Then, every time I get a new frame to encode I pass it to this function (the frames are being decoded via ffmpeg from another mp4 file which Quicktime plays just fine):

// If frame == nullptr then we're done and we're just flushing the encoder
// otherwise encode an actual frame
void FfmpegEncodingFrameSink::EncodeAndWriteFrame(
    const AVFrame* frame) {
  auto ret = avcodec_send_frame(codec_ctxt_.get(), frame);
  if (ret < 0) {
    throw FfmpegException("Error encoding the frame.");
  }

  AVPacket enc_packet;
  enc_packet.data = nullptr;
  enc_packet.size = 0;
  av_init_packet(&enc_packet);

  do {
    ret = avcodec_receive_packet(codec_ctxt_.get(), &enc_packet);
    if (ret ==  AVERROR(EAGAIN)) {
      CHECK(frame != nullptr);
      break;
    } else if (ret ==  AVERROR_EOF) {
      CHECK(frame == nullptr);
      break;
    } else if (ret < 0) {
      throw FfmpegException("Error putting the encoded frame into the packet.");
    }

    assert(ret == 0);
    enc_packet.stream_index = 0;

    LOG(INFO) << "Writing packet to stream.";
    av_interleaved_write_frame(format_ctxt_.get(), &enc_packet);
    av_packet_unref(&enc_packet);
  } while (ret == 0);
}

Finally, in my destructor I close everything up like so:

FfmpegEncodingFrameSink::~FfmpegEncodingFrameSink() {
  // Pass a nullptr to EncodeAndWriteFrame so it flushes the encoder
  EncodeAndWriteFrame(nullptr);
  // write mp4 trailer
  av_write_trailer(format_ctxt_.get());
}

If I run this passing n frames to EncodeAndWriteFrame line LOG(INFO) << "Writing packet to stream."; gets run n times indicating the n packets were written to the stream. But ffprobe always shows only n - 1 frames int he video. And the final video doesn't play on quicktime.

What am I doing wrong??

Harry Harry · Accepted Answer · 2019-12-05T14:18:29

Sorry for the delay but as i just had the same problem and noticed that this question deserves an answer, here how i solved this. Up in front, the Problem only occured for me when using mov, mp4, 3gp as format. It worked frame accurate when using e.g. avi format. When i wrote uncompressed video frames to the container, i saw that the avi and mov had the same count of frames stored but the mov obviously had some problem in it's header.

Counting the number of frames in the mov using header metadata showed one frame is missing:

ffprobe -v error -count_frames -select_streams v:0 -show_entries stream=nb_read_frames -of default=nokey=1:noprint_wrappers=1 c:\temp\myinput.mov

While ignoring the index showed the correct number of frames:

-ignore_editlist 1

The solution for me was, set the timebase to the AVStream->CodeContext of the video stream.

The code above attempts to do this in this line:

out_stream_video->time_base = codec_context->time_base;

But the problem is that the posted code above does not expose the function GetCodecContext so we do not know if the time_base is correctly set for "codec_context". So it is my believe that the author's problem was that his function GetCodecContext did not set the time_base correctly.

C++ ffmpeg video missing frames and won't play in Quicktime

1 Answers