ffmpeg based multi threaded c++ application fails on decoding

Question

I am using remuxing example from ffmpeg sources as reference. I wrote a multi-threaded application based on boost threads to perform a codec copy and remux using ffmpeg API. That works fine . The problem arises when I try to decode the frame

"

 ret = avcodec_decode_video2(dec_ctx, frame, &got_frame, &pkt);
            if (ret < 0) {
                av_log(NULL, AV_LOG_ERROR, "Error decoding video %s\n",av_make_error_string(errorBuff,80,ret));
                return -1;
            }"

I need the decoded frame to convert it to Opencv Mat object. For a single instance this code works fine. But as soon as I run multiple threads I start getting decoding errors like these

   left block unavailable for requested intra mode at 0 0
[h264 @ 0x7f9a48115100] error while decoding MB 0 0, bytestream 1479
[h264 @ 0x7f9a480825e0] number of reference frames (0+2) exceeds max (1; probably corrupt input), discarding one
[h264 @ 0x7f9a480ae680] error while decoding MB 13 5, bytestream -20
[h264 @ 0x7f9a48007700] number of reference frames (0+2) exceeds max (1; probably corrupt input), discarding one
[h264 @ 0x7f9a48110340] top block unavailable for requested intra4x4 mode -1 at 31 0
[h264 @ 0x7f9a48110340] error while decoding MB 31 0, bytestream 1226
[h264 @ 0x7f9a48115100] number of reference frames (0+2) exceeds max (1; probably corrupt input), discarding one
[h264 @ 0x7f9a480825e0] top block unavailable for requested intra4x4 mode -1 at 4 0
[h264 @ 0x7f9a480825e0] error while decoding MB 4 0, bytestream 1292
[h264 @ 0x7f9a480ae680] number of reference frames (0+2) exceeds max (1; probably corrupt input), discarding one

All variables used by ffmpeg api are declared local to the thread function. I am not sure how ffmpeg frame allocs or context allocs work.

any help in making the decoding process multi-threaded ?

Update: I have included ff_lockmgr

static int ff_lockmgr(void **mutex, enum AVLockOp op)
{
   pthread_mutex_t** pmutex = (pthread_mutex_t**) mutex;
   switch (op) {
   case AV_LOCK_CREATE:
      *pmutex = (pthread_mutex_t*) malloc(sizeof(pthread_mutex_t));
       pthread_mutex_init(*pmutex, NULL);
       break;
   case AV_LOCK_OBTAIN:
       pthread_mutex_lock(*pmutex);
       break;
   case AV_LOCK_RELEASE:
       pthread_mutex_unlock(*pmutex);
       break;
   case AV_LOCK_DESTROY:
       pthread_mutex_destroy(*pmutex);
       free(*pmutex);
       break;
   }
   return 0;
}

and initialized it as well "av_lockmgr_register(ff_lockmgr);" Now the video is being decoded in all threads BUT the images saved from the decoded frame using FFMPEG AVFrame to OpenCv Mat conversion and imwrite results in garbled (mixed) frame. Part of the frame is from one camera and rest is from another or the image doesnt make any sense at all.

Something in the code is wrong. Extract a minimal example as per posting guidelines to get help (and to help yourself). — Ulrich Eckhardt
Are you suggesting you're setting up "boost threads" yourself and calling avcodec_decode_video2() on the same AVCodecContext from multiple threads simultaneously? That's not how ffmpeg multithreading works, and it cannot work like that because you need to tune the thread locking so inter prediction from frame to frame works. Let ffmpeg deal with it (it does so automatically) and don't do any additional threading setup yourself. — Ronald S. Bultje
No, I am not calling avcodec_decode_video2() on same AVCodecContext. I have multiple boost threads each with localised variables for AV* library usage AVCodecContext n such. The threads do not share any data between them. Each has a separate video input and a separate video output. — Muhammad Ali

George Y. George Y. · Accepted Answer · 2016-01-23T03:37:24

Not every format decoder supports multiple threads, and even for the decoders which support it, it might not be supported for a particular file.

For example, consider a MPEG4 file with a single keyframe at the beginning, followed by P frames. In this case every next frame depends on previous, and using multiple threads would not likely produce any benefits.

In my app I had to disable multithreaded encoders because of that.

ffmpeg based multi threaded c++ application fails on decoding

1 Answers