Process AVFrame using opencv mat causing encoding error

Question

I'm trying to decode a video file using ffmpeg, grab the AVFrame object, convert it to opencv mat object, do some processing then convert it back to AVFrame object and encode it back to a video file.

Well, the program can run, but it produces bad result.

I Keep getting errors like "top block unavailable for requested intra mode at 7 19", "error while decoding MB 7 19, bytestream 358", "concealing 294 DC, 294AC, 294 MV errors in P frame" etc.

And the result video got glithes all over it. like this, enter image description here

I'm guessing it's because my AVFrame to Mat and Mat to AVFrame methods, and here they are

//unspecified function
temp_rgb_frame = avcodec_alloc_frame(); 
int numBytes = avpicture_get_size(PIX_FMT_RGB24, width, height); 
uint8_t * frame2_buffer = (uint8_t *)av_malloc(numBytes * sizeof(uint8_t)); 
avpicture_fill((AVPicture*)temp_rgb_frame, frame2_buffer, PIX_FMT_RGB24, width, height);

void CoreProcessor::Mat2AVFrame(cv::Mat **input, AVFrame *output)
{
    //create a AVPicture frame from the opencv Mat input image
    avpicture_fill((AVPicture *)temp_rgb_frame,
        (uint8_t *)(*input)->data,
        AV_PIX_FMT_RGB24,
        (*input)->cols,
        (*input)->rows);

    //convert the frame to the color space and pixel format specified in the sws context

    sws_scale(
        rgb_to_yuv_context, 
        temp_rgb_frame->data,
        temp_rgb_frame->linesize,
        0, height, 
        ((AVPicture *)output)->data, 
        ((AVPicture *)output)->linesize);

    (*input)->release();

}

void CoreProcessor::AVFrame2Mat(AVFrame *pFrame, cv::Mat **mat)
{
    sws_scale(
        yuv_to_rgb_context, 
        ((AVPicture*)pFrame)->data, 
        ((AVPicture*)pFrame)->linesize, 
        0, height, 
        ((AVPicture *)temp_rgb_frame)->data,
        ((AVPicture *)temp_rgb_frame)->linesize);

    *mat = new cv::Mat(pFrame->height, pFrame->width, CV_8UC3, temp_rgb_frame->data[0]);
}

void CoreProcessor::process_frame(AVFrame *pFrame)
{
    cv::Mat *mat = NULL;
    AVFrame2Mat(pFrame, &mat);
    Mat2AVFrame(&mat, pFrame);
}

Am I doing something wrong with the memory? Because if I remove the processing part, just decode and then encode the frame, the result is correct.

motivation for (*input)->release();? and where is temp_rgb_frame declared? — UmNyobe
temp_rgb_frame is initialized before main decoding part starts, like this temp_rgb_frame = avcodec_alloc_frame(); int numBytes = avpicture_get_size(PIX_FMT_RGB24, width, height); uint8_t * frame2_buffer = (uint8_t *)av_malloc(numBytes * sizeof(uint8_t)); avpicture_fill((AVPicture*)temp_rgb_frame, frame2_buffer, PIX_FMT_RGB24, width, height); And I want to release the mat so I called release method after convert it back to a AVFrame — user2789801
My fault, in my code ,I made a mistake at the temp_rgb_frame initialization part, I set the pixel format wrong, it should be PIX_FMT_RGB24, but I set it to PIX_FMT_YUV420P, now it works! Thanks UmNyobe!!! — user2789801
note that avpicture_fill doesnt perform a deep copy of the data. It only setup pointers and linesize base on the param. which means if you delete the original data, you are operating on released memory. — UmNyobe

user2789801 user2789801 · Accepted Answer · 2014-11-13T14:23:10

Well, it turns out I made a mistake at the initialization of temp_rgb_frame,if should be like this,

temp_rgb_frame = avcodec_alloc_frame();
int numBytes = avpicture_get_size(PIX_FMT_RGB24, width, height);
uint8_t * frame2_buffer = (uint8_t *)av_malloc(numBytes * sizeof(uint8_t));
avpicture_fill((AVPicture*)temp_rgb_frame, frame2_buffer, PIX_FMT_RGB24, width, height);

Process AVFrame using opencv mat causing encoding error

1 Answers