I am trying to reconstruct a scene in 3D based on iamge features and projection matrices (computed through feature matching and essential matrix computation) of two cameras, all of this in OpenCV. I am encountering a strange problem with this.
Mainly, the accuracy of the triangulation of points and scene reconstruction is extremely sensitive to the R matrix. To test this, I ran a test with simulated object points propagated through two projection matrices (to simulate the stereo). The object points are a 6x6 square; x:{0 to 1 in steps of 0.2}, y:{0 to 1 in steps of 0.2}, z:{all 10}. P1 is [I|0] in both cases, P2 is [R|t], where:
R1 = 0.9995726521705084, -0.003399231961472978, -0.02903374338399459,
0.003763143573699675, 0.9999149324186841, 0.01248866192120443,
0.02898882169490978, -0.01259258306347518, 0.9995004127405501
t1 = [0.999, 0.01, 0.01]
R2 = 0.9989054656616956, -0.005509540027623473, 0.04644906500539195,
0.004867423001511094, 0.9998911828244373, 0.01392590044744823,
-0.04652073585526645, -0.01368457082381193, 0.9988235848521261
t2 = [0.999, 0.02, 0.01]
I thought these two R matrices were close enough to be considered as accurate. But the triangulation results seem to think otherwise..
With R as identity: (close to perfect reconstruction)
[5.0598763e-05, -0.0063825068, 9.9969149;
4.8001177e-05, 0.19379893, 9.9967384;
4.6123398e-05, 0.39403415, 9.9965773;
4.3485958e-05, 0.59420204, 9.9964066;
4.0845167e-05, 0.79436392, 9.9962406;
3.8896389e-05, 0.99458051, 9.996089;
0.19970703, -0.0063951053, 9.9936533;
0.19969735, 0.19372027, 9.9934254;
0.19968878, 0.39388725, 9.9932156;
0.19967939, 0.59398526, 9.9929981;
0.19967027, 0.79407561, 9.9927874;
0.19966206, 0.99421865, 9.99259;
0.39928076, -0.0064075897, 9.9915905;
0.39926201, 0.19366568, 9.991312;
0.39924476, 0.39378849, 9.9910526;
0.39922675, 0.5938406, 9.9907885;
0.39920929, 0.79388332, 9.9905319;
0.39919299, 0.99397665, 9.9902887;
0.59869993, -0.0064199544, 9.9883261;
0.59867001, 0.19358718, 9.987998;
0.59864205, 0.39364177, 9.987689;
0.59861356, 0.593624, 9.9873781;
0.59858596, 0.79359502, 9.9870768;
0.5985598, 0.99361473, 9.9867878;
0.79777306, -0.0064321696, 9.9838686;
0.79773003, 0.19348493, 9.98349;
0.79768944, 0.39344737, 9.9831314;
0.79764849, 0.59333575, 9.9827728;
0.79760903, 0.79321122, 9.9824266;
0.79757106, 0.99313337, 9.9820919;
0.99702829, -0.0064441776, 9.9818068;
0.99697012, 0.19343086, 9.9813776;
0.99691486, 0.39334908, 9.9809694;
0.99685961, 0.59319156, 9.9805641;
0.99680638, 0.79301941, 9.9801722]
Using R1|t1:
[0.0006948277, -0.057961587, 7.7867656;
0.00076079171, 0.097548366, 7.7840567;
0.00082782045, 0.25295791, 7.781354;
0.00089472742, 0.40817291, 7.7786431;
0.00096202898, 0.56324077, 7.7759318;
0.0010304272, 0.71820831, 7.7732263;
0.1562658, -0.058364984, 7.7859602;
0.15625301, 0.097173139, 7.7831898;
0.15624109, 0.25260818, 7.7804236;
0.15622856, 0.407846, 7.7776465;
0.15621613, 0.5629338, 7.7748661;
0.15620455, 0.71791828, 7.772089;
0.31177881, -0.058764182, 7.7844906;
0.31168443, 0.096788943, 7.7816563;
0.31159076, 0.25223625, 7.7788253;
0.31149602, 0.40748379, 7.7759824;
0.3114011, 0.5625785, 7.7731342;
0.31130677, 0.71756685, 7.7702885;
0.46710756, -0.05914975, 7.7809157;
0.46692884, 0.096376404, 7.7780185;
0.46675056, 0.25179401, 7.7751222;
0.46657094, 0.40700936, 7.7722144;
0.46639091, 0.5620693, 7.7693019;
0.46621126, 0.71701992, 7.7663894;
0.6220336, -0.059520967, 7.7752504;
0.62176758, 0.095936403, 7.7722869;
0.62150216, 0.25128254, 7.7693276;
0.62123501, 0.40642393, 7.766356;
0.62096751, 0.56140751, 7.763382;
0.6207003, 0.71627909, 7.7604065;
0.77683568, -0.0598915, 7.7696557;
0.77647918, 0.095498636, 7.7666249;
0.77612352, 0.25077474, 7.7635999;
0.77576625, 0.4058437, 7.7605648;
0.77540869, 0.56075233, 7.7575278]
Using R2|t2:
[0.00092721562, -0.13777997, 18.777367;
0.00076921738, 0.2374147, 18.73959;
0.00061207288, 0.61111385, 18.701971;
0.00045474528, 0.9831, 18.664478;
0.0002978587, 1.3534971, 18.627134;
0.00014151275, 1.7224245, 18.58993;
0.37676802, -0.13919897, 18.809832;
0.3757931, 0.23646803, 18.771765;
0.37482125, 0.61062753, 18.733831;
0.37385073, 0.98306084, 18.695993;
0.37288278, 1.3538899, 18.658278;
0.37191704, 1.7232322, 18.62067;
0.75461411, -0.14074895, 18.859949;
0.75280511, 0.23574051, 18.821489;
0.75099939, 0.61070609, 18.783117;
0.74919492, 0.98392648, 18.744808;
0.74739331, 1.3555217, 18.706583;
0.74559367, 1.7256067, 18.668427;
1.1350268, -0.14237344, 18.919434;
1.132362, 0.23512313, 18.880507;
1.1296982, 0.61107594, 18.841614;
1.127033, 0.98526078, 18.802734;
1.1243687, 1.357795, 18.763895;
1.1217037, 1.7287903, 18.725073;
1.5183519, -0.14407407, 18.988445;
1.5148053, 0.23461713, 18.948969;
1.5112541, 0.61174142, 18.909468;
1.5076962, 0.98707068, 18.869923;
1.5041342, 1.3607191, 18.830359;
1.5005655, 1.7327945, 18.790751;
1.9069529, -0.14595015, 19.080332;
1.9024866, 0.2343885, 19.040174;
1.8980062, 0.6131314, 18.999918;
1.8935097, 0.99004531, 18.959545;
1.889001, 1.3652413, 18.91909]
The accuracy of t is not causing any problems of this magnitude. Surely, a ~0.01 differences in the rotation shouldn't cause such bad reconstruction.. Can anyone shed some light on what might be going wrong here? I tried two methods of triangulation, one that's inbuilt into opencv (cv::triangulatePoints()) and another an iterative least squares method explained in the Hartley and Zisserman book. My pipeline is this: input image points -> homogenize using K matrices -> triangulate -> "unhomogenize" obtained 4d points to obtain 3d. Thanks for your time!
