Opencv , transforming image by affine transformation

Question

I want to transform the first image to second image, I think maybe it is affine transformation. My problems and questions are as follows:

(1). As I have mentioned above, I think the transform is affine transformation. So the first step is to find three pairs of corresponding points by clicking three corner points in the first image along clockwise direction(return coordinates from mouse callback function) and set their corresponding points as specific coordinates(the distances between each corner point are known). And the second step is to use getAffineTransform() and warpAffine() methods to implement affine transformation. But this approach have turned out to be not good enough(see the third image), so is there any idea to improve the result?

My code is shown as follows:

#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc/imgproc.hpp>
#include "opencv2/features2d/features2d.hpp"

#include <stdio.h>
#include <vector>
#include <iostream>
#include <fstream>

using namespace std;
using namespace cv;


Mat org;
int n=0;
vector<Point> capturePoint;

void on_mouse(int event,int x,int y,int flags,void *ustc)
{
    Point pt;
    char coordinateName[16];

    if (event == CV_EVENT_LBUTTONDOWN)
    {
        pt = Point(x,y);
        cout<<x<<" "<<y<<endl;
        capturePoint.push_back(pt);
        n++;

        circle(org,pt,2,Scalar(255,0,0,0),CV_FILLED,CV_AA,0);
        sprintf(coordinateName,"(%d,%d)",x,y);
        putText(org,coordinateName,pt,FONT_HERSHEY_SIMPLEX,0.5,Scalar(0,0,0,255),1,8);

        //imshow("org",org);

        if(n>=4)
        {
            imshow("org",org);
            cvDestroyAllWindows();
        }
    }
}

int main()
{
    org = imread("1-3.jpg",1);

    namedWindow("org",1);
    setMouseCallback("org",on_mouse,0);// mouse callback;

    imshow("org",org);
    waitKey(0);
    //cout<<capturePoint.size()<<endl;

    //three pairs of corresponding points;

    Point2f srcPoint[3];
    Point2f dstPoint[3];
    srcPoint[0]=capturePoint[0];//mouse click along clockwise direction;
    srcPoint[1]=capturePoint[1];
    srcPoint[2]=capturePoint[2];
    //srcPoint[3]=capturePoint[3];

    dstPoint[0]=Point(0,0);//distances between each corner point are known;
    dstPoint[1]=Point(640,0);//width=320,height=220;
    dstPoint[2]=Point(640,440);
    //dstPoint[3]=Point(0,220);

    Mat warpDst=Mat(org.rows, org.cols, org.type());
    //Mat warpMat = findHomography( srcPoint, dstPoint, 0 );
    Mat warpMat = getAffineTransform(srcPoint,dstPoint);
    warpAffine(org,warpDst,warpMat,org.size());//affine transformation;
    imshow("Warp",warpDst);

    waitKey(0);

    return 0;
}

(2). I would like to implement the transformation without manual intervention, i.e. selecting points manually is not required, so I want to use the centers of ellipses in the first image and their corresponding points to implement affine transformation. My question is that if the centers of ellipses can be used to implement affine transformation, and if can be, how to detect the centers of ellipses.

(3). I have used canny method to get the largest contour, and applied cornerHarris to detect corner points from the largest contour. I found that each corner may have some candidate points that lie close to each other. So how to get the four "ture" corner points?

Answer 1

I played with your image using cv2 (python binding). It's still not perfect but you can play with the parameters to improve further. Smaller k improves corner detection. Threshold is also important.

import cv2
from pylab import *

fp = "qkwWV.jpg"

img = cv2.imread(fp)
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

# edge and corner detection using cornerHarris
blockSize = 16
ksize  = 7
k = 0.005
corners = cv2.cornerHarris(gray, blockSize, ksize, k)
cv2.normalize( corners, corners, 0, 255, cv2.NORM_MINMAX, cv2.CV_32F, None)

# play with threshold until only 4 corners are left.
figure()
thrs = 173
imshow(corners>thrs, cmap=cm.gray)

# get the center coordinates of 4 groups using k-means
y,x = np.where(corners>thrs)
candidates = np.array(zip(x,y)).astype(np.float32)

term_crit = (cv2.TERM_CRITERIA_EPS, 30, 0.1)
flags = cv2.KMEANS_RANDOM_CENTERS
ret, labels, centers = cv2.kmeans(candidates, 4, term_crit, 10, flags)

# draw cross haris on the original image
centers = centers.astype(np.int)
L = 40
for _x, _y in centers: 
    cv2.line(img, (_x-L, _y), (_x+L, _y), (255,0,0), 3)
    cv2.line(img, (_x, _y-L), (_x, _y+L), (255,0,0), 3)

figure()
imshow(img)
show()