Obtaining matching image frames from videos with two different frame rates

0
votes

I am in the process of converting videos to images in python3.6 (i.e. cut videos to get images)

I have two types of videos The first one is a RGB video recorded from Real sense D435i with a frame rate of 30 (i.e. fps= 30) The second one is a thermal-IR video recorded from Flir Adas camera with a frame rate of 9 (It was originally a frame stream file which i converted to images and formulated it into a video with fps of 9 using python3-cv2). The video formats are in mp4 and avi respectively (though I have converted the avi to mp4 and tested it out as well). They are equal in length.

I am trying to create a matched image pair from the thermal and IR videos. However, when I seem to cut them using the same frame rate, it seems like they don't match (but differ by 5-6 images).

I have about 200 ish videos so it is very time consuming and difficult for me to track them down one by one.

Any ideas on how I can get this dataset to make it a paired one?

Many thanks

2
image-processingdeep-learningdatasetvideo-processingframe-rate
Please Edit your post, add sample videos from your data-set (if possible) and the code that you did try, in order to form the minimal-reproducible-example – Bilal

2 Answers

0
votes

You might be able to recreate the 30 fps video as a matching 9 fps video by matching up the timestamps of the two videos. OpenCV lets you specify the framerate in the VideoWriter (I think you knew this, but just to be sure). OpenCV reports the timestamp of the current video with

cv2.VideoCapture.get(cv2.CAP_PROP_POS_MSEC);

Grab one frame and timestamp from the FLIR camera and then keep grabbing frames and timestamps from the color camera until the timestamp catches up or passes the FLIR timestamp. Then write the color frame to the vidwriter. So long as both of the videos had a consistent framerate and they both started/stopped recording at the same time then this should see both videos matched up as closely as possible.

0
votes

What about just converting the 9 fps videos to 30 fps with a simple ffmpeg script over the list of files:

ffmpeg -i source9fps.mp4 -r 30 -c:v libx264 -c:b 10M out30fps.mp4

#-c:b ... bitrate

or:

ffmpeg -i source9fps.mp4 -r 30 -c:v copy out30fps.mp4

(Without reencoding, very fast)

30 vs 9 fps is a nasty combination though, there will be a skew and I'm not sure about the exact conversion algorithm (some blending might be better due to the skew at frames with a big temporal mismatch).

It'd be better if the second video was recorded at 5, 10 or 15 fps.