Sfm, reconstruction from stereo image

Question

I need your help. I have to rebuild markers in 3D space from stereo image. In my case, I would like to reconstruct the markers using an uncalibrated method.

I shoot 2 photos and sign the markers manually for now.

import cv2
import numpy as np
from matplotlib import pyplot as plt
from scipy import linalg


img1 =  cv2.imread('3.jpg',0)


img2 = cv2.imread('4.jpg',0)


pts1 = np.array([(1599.6711229946527, 1904.8048128342245), (1562.131016042781, 1734.4304812834225), (1495.7139037433158, 1295.5), 
     (2373.5748663101604, 1604.4839572192514), (2362.0240641711234, 2031.8636363636363), (2359.136363636364, 2199.3502673796793), 
     (2656.5695187165775, 1653.5748663101604), (2676.7834224598937, 1506.302139037433), (2740.312834224599, 1026.9438502673797), 
     (1957.745989304813, 807.4786096256685)],dtype='float64')

pts2 = np.array([(1579.457219251337, 1899.0294117647059), (1539.0294117647059, 1737.3181818181818),
     (1472.612299465241, 1307.0508021390374), (2315.8208556149734, 1633.3609625668448),
     (2298.4946524064176, 2054.9652406417113), (2301.3823529411766, 2190.687165775401), 
     (2630.5802139037432, 1670.9010695187167), (2642.131016042781, 1538.066844919786),
     (2711.4358288770054, 1076.0347593582887), (1949.0828877005351, 842.1310160427806)],dtype='float64')

subsequently I find the fundamental matrix

F, mask = cv2.findFundamentalMat(pts1,pts2,cv2.FM_7POINT)

and print the result from cv2.computeCorrespondEpilines

link

it would seem to work well!

I have the camera matrix, previously calibrated with a chessboard, following the tutorial on the opencv website

mtx=np.array([[3.19134206e+03, 0.00000000e+00, 2.01707613e+03],
       [0.00000000e+00, 3.18501724e+03, 1.54542273e+03],
       [0.00000000e+00, 0.00000000e+00, 1.00000000e+00]])

extract the Essential matrix , following what is reported in the book Hartley and Zisserman

E = K.t() * F * K

E = mtx.T * F * mtx

I decomposed this matrix to find the rotation and translation matrices

R1, R2, T = cv2.decomposeEssentialMat(E)
kr= np.dot(mtx,R1)
kt= np.dot(mtx,T)
projction2=np.hstack((kr,kt))
projction1 = np.array([[1, 0, 0, 0], [0, 1, 0, 0], [0, 0, 1, 0]])

obtaining the projection matrices. P1 is the first matrix, which as always described in the above book is P1 = [I | 0] the second matrix is P2 = K [ R | t ]

now I used the following code to go back to the triangulation of the points

points4D = cv2.triangulatePoints(projction1, projction2, pts1.T, pts2.T)

I convert the homogeneous coordinates into Cartesian and the result is this:

coordinate_eucl= cv2.convertPointsFromHomogeneous(points4D.T)

coordinate_eucl=coordinate_eucl.reshape(-1,3)
x,y,z=coordinate_eucl.T

from mpl_toolkits.mplot3d import Axes3D
import matplotlib.pyplot as plt

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.scatter(x, y, z, c='r', marker='o')
ax.set_xlabel('X Label')
ax.set_ylabel('Y Label')
ax.set_zlabel('Z Label')
plt.show()

link

what am I wrong?

thx

unlut unlut · Accepted Answer · 2018-09-06T14:19:25

It is good to check each step individually. (You may want to look 4th step first)
1- First, you said you calibrated camera previously. How much reprojection error you got? Did you do any checks to validate success of your calibration. I also assume both your cameras mostly identical.
2- If the fundamental matrix you found is correct (make sure your point lists have same order for both list by the way), it should satisfy epipolar constraint p' F p = 0 where p' is the point in right view and p is point in left view (homogeneous pixel coordinates). Although they will not be exactly 0, should be close to 0. This equation must hold for all point correspondences. If it does not, try using CM_FM_RANSAC or skip to step 3.
3- Check whether you can directly calculate essential matrix with opencv function. Also, a similar equation must hold for essential matrix.
4- OpenCV decomposeEssentialMat function returns two possible rotation matrices and there are two possible translations (so total 4 possible R T combination). Try testing all of them. If you can get correct solution with one of 4 combinations, I will edit my answer to include how to get correct combination.
If your fundamental / essential matrix calculations are correct and problem still occurs, please let me know.

Sfm, reconstruction from stereo image

1 Answers