I need to compute the frame differences between a source video and a compressed one. For now I'm using OpenCV with Java, by extracting each frame and doing a simple difference, but it's quite slow (working a 0.5 fps, meaning that a 500 frames video will take more than 15 mins) so I was thinking to move to FFMPEG.
FFMPEG feels a lot faster (everything's done under 1 minute) but it has one big issue that makes the results useless: when compressing the source file, done with FFMPEG too, an extra gray frame is added at the beginning and this fakes the results because different frames are compared.
This is what I'm doing now (knowing that the extra frame messes it all):
ffmpeg -y -i src.avi -i compressed.avi -filter_complex "blend=all_mode=difference,hue=s=0" -c:v libx264 -crf 18 -c:a copy difference.avi
To fix the frame issue I was trying to remove the first frame by re-encoding the compressed video with this command
ffmpeg -y -ss 0.02 -i compressed.mpg -an -f mpeg2video compressed-cut.mpg"
(Note that -ss is 0.02 because it's a 50 fps video, so I did 1/FPS as suggested here)
But I get this response
Output file is empty, nothing was encoded (check -ss / -t / -frames parameters if used)
So, finally, the question is: since extracting all the frames and then compute differences with OpenCV is really slow, how can I use FFMPEG to produce a video containing the difference between two sources while keeping in mind that one of them has an extra frame at the beginning?
EDIT: I wanted to avoid posting endless console outputs but since you asked for it, here we go.
1) Encoding
Input
ffmpeg -i "720p50_mobcal_ter.avi" -an -f mpeg2video -y "720p50_mobcal_ter.mpg"
Output
ffmpeg version N-76684-g1fe82ab Copyright (c) 2000-2015 the FFmpeg developers
built with gcc 5.2.0 (GCC)
configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libdcadec --enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libzimg --enable-lzma --enable-decklink --enable-zlib
libavutil 55. 6.100 / 55. 6.100
libavcodec 57. 15.100 / 57. 15.100
libavformat 57. 14.100 / 57. 14.100
libavdevice 57. 0.100 / 57. 0.100
libavfilter 6. 15.100 / 6. 15.100
libswscale 4. 0.100 / 4. 0.100
libswresample 2. 0.101 / 2. 0.101
libpostproc 54. 0.100 / 54. 0.100
Input #0, avi, from '720p50_mobcal_ter.avi':
Metadata:
encoder : Lavf57.14.100
Duration: 00:00:10.08, start: 0.000000, bitrate: 552974 kb/s
Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 1280x720, 554059 kb/s, SAR 1:1 DAR 16:9, 50 fps, 50 tbr, 50 tbn, 50 tbc
Output #0, mpeg2video, to '720p50_mobcal_ter.mpg':
Metadata:
encoder : Lavf57.14.100
Stream #0:0: Video: mpeg2video, yuv420p, 1280x720 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 50 fps, 50 tbn, 50 tbc
Metadata:
encoder : Lavc57.15.100 mpeg2video
Stream mapping:
Stream #0:0 -> #0:0 (rawvideo (native) -> mpeg2video (native))
Press [q] to stop, [?] for help
frame= 41 fps=0.0 q=31.0 size= 984kB time=00:00:00.78 bitrate=10330.5kbits/frame= 80 fps= 78 q=31.0 size= 1323kB time=00:00:01.56 bitrate=6948.1kbits/frame= 124 fps= 80 q=31.0 size= 1725kB time=00:00:02.44 bitrate=5790.0kbits/frame= 168 fps= 81 q=31.0 size= 2084kB time=00:00:03.32 bitrate=5142.8kbits/frame= 212 fps= 81 q=31.0 size= 2482kB time=00:00:04.20 bitrate=4841.4kbits/frame= 255 fps= 82 q=31.0 size= 2840kB time=00:00:05.06 bitrate=4597.2kbits/frame= 296 fps= 82 q=31.0 size= 3133kB time=00:00:05.88 bitrate=4364.5kbits/frame= 338 fps= 82 q=24.8 size= 3453kB time=00:00:06.72 bitrate=4209.2kbits/frame= 382 fps= 82 q=31.0 size= 3723kB time=00:00:07.60 bitrate=4013.4kbits/frame= 426 fps= 83 q=31.0 size= 4005kB time=00:00:08.48 bitrate=3869.1kbits/frame= 470 fps= 83 q=24.8 size= 4276kB time=00:00:09.36 bitrate=3742.5kbits/frame= 504 fps= 83 q=31.0 Lsize= 4469kB time=00:00:10.06 bitrate=3639.3kbits/s
video:4469kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000000%
This adds the extra grey frame at the beginning, it just duplicates the first one
2) Removing first frame
Input
ffmpeg -y -i "720p50_mobcal_ter.mpg" -an -f mpeg2video -vf select=gte(n\,1) "CUT-720p50_mobcal_ter.mpg"
Output
ffmpeg version N-76684-g1fe82ab Copyright (c) 2000-2015 the FFmpeg developers
built with gcc 5.2.0 (GCC)
configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libdcadec --enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libzimg --enable-lzma --enable-decklink --enable-zlib
libavutil 55. 6.100 / 55. 6.100
libavcodec 57. 15.100 / 57. 15.100
libavformat 57. 14.100 / 57. 14.100
libavdevice 57. 0.100 / 57. 0.100
libavfilter 6. 15.100 / 6. 15.100
libswscale 4. 0.100 / 4. 0.100
libswresample 2. 0.101 / 2. 0.101
libpostproc 54. 0.100 / 54. 0.100
Input #0, mpegvideo, from '720p50_mobcal_ter.mpg':
Duration: N/A, bitrate: N/A
Stream #0:0: Video: mpeg2video (Main), yuv420p(tv), 1280x720 [SAR 1:1 DAR 16:9], max. 104857 kb/s, 50 fps, 50 tbr, 1200k tbn, 100 tbc
Output #0, mpeg2video, to 'CUT-720p50_mobcal_ter.mpg':
Metadata:
encoder : Lavf57.14.100
Stream #0:0: Video: mpeg2video, yuv420p, 1280x720 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 50 fps, 50 tbn, 50 tbc
Metadata:
encoder : Lavc57.15.100 mpeg2video
Stream mapping:
Stream #0:0 -> #0:0 (mpeg2video (native) -> mpeg2video (native))
Press [q] to stop, [?] for help
frame= 255 fps=0.0 q=31.0 size= 2781kB time=00:00:05.10 bitrate=4467.3kbits/frame= 503 fps=0.0 q=31.0 Lsize= 4415kB time=00:00:10.08 bitrate=3588.5kbits/s
video:4415kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000000%
3) Frame difference
Input
ffmpeg -y -i "720p50_mobcal_ter.avi" -i "CUT-720p50_mobcal_ter.mpg" -filter_complex "blend=all_mode=difference,hue=s=0" -c:v libx264 -crf 18 -c:a copy "DIFF-720p50_mobcal_ter.mpg"
Output
ffmpeg version N-76684-g1fe82ab Copyright (c) 2000-2015 the FFmpeg developers
built with gcc 5.2.0 (GCC)
configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libdcadec --enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libzimg --enable-lzma --enable-decklink --enable-zlib
libavutil 55. 6.100 / 55. 6.100
libavcodec 57. 15.100 / 57. 15.100
libavformat 57. 14.100 / 57. 14.100
libavdevice 57. 0.100 / 57. 0.100
libavfilter 6. 15.100 / 6. 15.100
libswscale 4. 0.100 / 4. 0.100
libswresample 2. 0.101 / 2. 0.101
libpostproc 54. 0.100 / 54. 0.100
Input #0, avi, from '720p50_mobcal_ter.avi':
Metadata:
encoder : Lavf57.14.100
Duration: 00:00:10.08, start: 0.000000, bitrate: 552974 kb/s
Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 1280x720, 554059 kb/s, SAR 1:1 DAR 16:9, 50 fps, 50 tbr, 50 tbn, 50 tbc
Input #1, mpegvideo, from 'CUT-720p50_mobcal_ter.mpg':
Duration: N/A, bitrate: N/A
Stream #1:0: Video: mpeg2video (Main), yuv420p(tv), 1280x720 [SAR 1:1 DAR 16:9], max. 104857 kb/s, 50 fps, 50 tbr, 1200k tbn, 100 tbc
[libx264 @ 000002784dbeb980] using SAR=1/1
[libx264 @ 000002784dbeb980] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 AVX2 LZCNT BMI2
[libx264 @ 000002784dbeb980] profile High, level 3.2
[mpeg @ 000002784dbeaf20] VBV buffer size not set, using default size of 130KB
If you want the mpeg file to be compliant to some specification
Like DVD, VCD or others, make sure you set the correct buffer size
Output #0, mpeg, to 'D:\DOWNLOADS\TMP\Video TDI\AVI\DIFF-720p50_mobcal_ter.mpg':
Metadata:
encoder : Lavf57.14.100
Stream #0:0: Video: h264 (libx264), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], q=-1--1, 50 fps, 90k tbn, 50 tbc (default)
Metadata:
encoder : Lavc57.15.100 libx264
Stream mapping:
Stream #0:0 (rawvideo) -> blend:top
Stream #1:0 (mpeg2video) -> blend:bottom
hue -> Stream #0:0 (libx264)
Press [q] to stop, [?] for help
frame= 504 fps= 39 q=-1.0 Lsize= 32182kB time=00:00:10.04 bitrate=26258.5kbits/s
video:32061kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.377054%
[libx264 @ 000002784dbeb980] frame I:30 Avg QP:19.69 size:149974
[libx264 @ 000002784dbeb980] frame P:299 Avg QP:23.28 size: 69423
[libx264 @ 000002784dbeb980] frame B:175 Avg QP:24.48 size: 43280
[libx264 @ 000002784dbeb980] consecutive B-frames: 30.6% 69.4% 0.0% 0.0%
[libx264 @ 000002784dbeb980] mb I I16..4: 18.3% 51.4% 30.4%
[libx264 @ 000002784dbeb980] mb P I16..4: 0.6% 5.6% 2.4% P16..4: 35.9% 22.9% 15.6% 0.0% 0.0% skip:17.0%
[libx264 @ 000002784dbeb980] mb B I16..4: 0.2% 0.5% 0.3% B16..8: 49.5% 12.4% 5.6% direct:15.5% skip:16.1% L0:47.8% L1:42.1% BI:10.1%
[libx264 @ 000002784dbeb980] 8x8 transform intra:57.5% inter:38.5%
[libx264 @ 000002784dbeb980] coded y,uvDC,uvAC intra: 90.7% 0.0% 0.0% inter: 50.3% 0.0% 0.0%
[libx264 @ 000002784dbeb980] i16 v,h,dc,p: 32% 23% 35% 10%
[libx264 @ 000002784dbeb980] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 11% 11% 41% 7% 5% 6% 5% 6% 8%
[libx264 @ 000002784dbeb980] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 18% 14% 23% 8% 7% 7% 7% 7% 8%
[libx264 @ 000002784dbeb980] i8c dc,h,v,p: 100% 0% 0% 0%
[libx264 @ 000002784dbeb980] Weighted P-Frames: Y:33.8% UV:0.0%
[libx264 @ 000002784dbeb980] ref P L0: 58.1% 16.3% 14.2% 9.4% 2.0%
[libx264 @ 000002784dbeb980] ref B L0: 79.3% 20.7%
[libx264 @ 000002784dbeb980] kb/s:26056.02
The second command made everything work while the second one in the first part didn't (the one with the -ss option), so I could be quite happy about it but I'm not that sure if FFMPEG duplicates the first frame for every video or if it's just related to the one I'm using now, so it could be better to start off with a compressed video that has the same frame count of the original one.
So let's get to one final question: why does FFMPEG add a duplicated first frame at the beginning of the compressed video and how can I avoid that?
ffmpegwas adding the gray frame. Is this not true? Regardless, you should always show the complete console output with anyffmpegcommand you show. – llogan