Pixel level image registration / alignment?

3
votes

I'm trying to remove foreground from two images, here's a sample pair of images:

Image 1Image 2

As you can see, the Budweiser bottle is removed from the scene before the second shot is taken.

These photos were captured from a pinhole camera (iPhone), and, the tricky part is I'm hand-holding the camera, so it cannot be guaranteed that the images are perfectly aligned pixel by pixel, so a simple minus-threshold method will not work.

Then, I've decided to perform image registration using findHomography and warpPerspective from OpenCV, here's the result image:

Img 3

This image is warped with the matrix I've got from findHomography, it kind of improved the alignment quality, but still not that aligned so I can use a simple way to remove the foreground.

So, finally, I decided to implement a "fuzzy-minus" algorithm: for every pixel in image1, I'll look through a 7x7 neighbour in image2 (a 7 by 7 kernel?), using the minimal difference in grayscale as the result of minus, and threshold the result into binary image, here's what I've got:

Image4

And the result is still not good. Notice the white wholes in the bottle, this is produced due to similar grayscale value of foreground and background. So I'm not sure what to do now.

I can think of two ways to solve the problem, the first is to get a better aligned pair of images, and simply minus the pairs; the second is to use a more robust way to extract the foreground.

Can anyone give me some advice on how to deal with this kind of problem? I believe there should be some state-of-art algorithms or processing pipelines, but after googling around, I get nothing.

I'm using OpenCV with C++, it would be fantastic if you can tell me how to do it with these tools in hand.

Big big thanks in advance!

4
c++opencvimage-processingcomputer-vision
Try using a color difference instead of a grayscale difference, it would provide significantly more discrimination. - David Nilosek
@DavidNilosek, I've also tried HSV color space, what I've got is quite similar, I mean, the wholes are still there. - Void Main
Have you tried combining some feature detection with your algorithm..? Detect some corners using FAST or ORB and then do findHomography on the resultant image. Cheers. - scap3y
Thanks @scap3y, currently I'm using SURF feature points, the image above is aligned using SURF feature points. Will the type of feature points affect the result? - Void Main
Of course it will.! Try out various feature detectors and see which one works the best. Read up on the corresponding literature as well. You might want to look into some background subtraction techniques (my personal favourite is GMM but it really depends on the application).. HTH - scap3y

4 Answers

4
votes

The problem is not in your algorithm. You are having problem because the two scenes were not taken from exactly the same angle, as shown in the animation below. This slight difference highlight the edges in the subtraction.

enter image description here

You need a static camera in order to apply this approach.

1
votes

I suggest using mathematical morphology on the mask that you got to get rid of the artifacts. Try applying both opening and closing to get rid of the black and the white small regions.

Mathematical Morphology

Mathematical Morphology in opencv

The difference between the two picture is pretty huge, so you will need to use a large structure element, but I don't think you will be able to get rid of the shadow. For the two large strips in the background, you may try to use a horizontally shaped structure element as well.

Edit

Is it possible to produce a grayscale image instead of a binary image? if yes, you may try to experiment with the hat method for the shadow, but I am not sure about this point.

This is what I got using two different structure elements for closing THEN opening

Mat mask = imread("mask.jpg",CV_LOAD_IMAGE_GRAYSCALE);
morphologyEx(mask,mask,MORPH_CLOSE,getStructuringElement(CV_SHAPE_ELLIPSE,Size(50,10)));
morphologyEx(mask,mask,MORPH_OPEN,getStructuringElement(CV_SHAPE_ELLIPSE,Size(10,50)));
imshow("open",mask);
imwrite("maskopenclose.jpg",mask);

maskopenclose.jpg

0
votes

I would suggest optical flow for alignment and OpenCV's background subtraction algorithm:

http://docs.opencv.org/trunk/doc/tutorials/video/background_subtraction/background_subtraction.html

0
votes

I suggest that instead of using findHomography try using some of openCV's stereo correspondence functions: http://docs.opencv.org/modules/calib3d/doc/camera_calibration_and_3d_reconstruction.html

there is a sample code here: https://github.com/Itseez/opencv/blob/master/samples/cpp/stereo_calib.cpp