Feature tracking using optical flow

24
votes

I found a similar question in the forum. But the answer in there does not answer my question.

  • If I do feature detection (goodFeaturesToTrack) only once on the first image, and then use optical flow (calcOpticalFlowPyrLK) to track these features, the problem is: only the features detected on the first image can be tracked. When these features go beyond of the image, there would be no features to track.

  • If I do feature detection for every new image, the feature tracking is not stable, because the feature detected last time may not be detected this time.

I am using optical flow for 3D reconstruction. So I'm not interested in tracking what features, instead, I only care whether features in the field of view can be tracked stably. To summarize, my question is: how can I use optical flow to track old features, and in the meantime add new image features that come into the field of view and remove old features that go beyond the field of view?

3
c++opencvcomputer-vision
because it hasnt been mentioned in this old question, the python version of lkdemo does a little more than its c++ counterpart, check it out, may solve this for some: github.com/npinto/opencv/blob/master/samples/python/lkdemo.py - Stefan Karlsson

3 Answers

19
votes

Several approaches are possible. A good method goes like this:

  1. in Frame 1 detect N features, this is the Keyframe m=1
  2. in Frame k track the features by optical flow
  3. in Frame k if the number of successfully tracked features drops under N/2:
    • this frame is the keyframe m+1
    • compute the homography or the fundamental matrix describing the motion between the keyframes m and m+1
    • detect N features and discard the old ones
    • k := k+1 go to 2

In this method basically you estimate the camera motion between the last two keyframes.

Since you didn't mention what approach is used for 3D reconstruction I assumed either H or F are computed to estimated motion first. To estimate them accurately the baseline between the keyframes should be as wide as possible. In general, the best strategy is take into account the rough motion model of the camera. If the camera is held by hand a different strategy should be used compared to when the camera is fixed on the top of a car or a robot. I can provide a minimal working example in Python if that helps, let me know.

3
votes

Just for documentation purposes, there are several good GPU / C++ implementations of optical flow tracking. Your code may be better for your purposes, but if all you need is the output data of the tracks, consider checking any of the following sources: here, here, or here.

0
votes

There is another good way to add new features to the existing ones. You can pass a mask into cv::goodFeaturesToTrack(). So you would create a new Mat (same size as original image, type: CV_8UC1), set all pixels to 255 and draw each feature point as a black circle into this Mat. When you pass this mask into goodFeaturesToTrack() those black circles will be skipped by the function.

I would also recommend limiting the amount of features. Let's say you limit it to MAX_FEATURES = 300. You then check every cycle whether you have less tracks than MAX_FEATURES - z (e.g. z = 30). In case you do, search for up to z new features as stated above and add them to your feature-container.

Also note that you have to actively delete Features when tracking failed. You will therefore have to look at the status output of calcOpticalFlowPyrLK.