How does 2d to 3d projection work when depth is taken from a central point?

Question

I'm looking at 2d (depth / rgbd) to 3d point cloud projection. Having spent the past few days reading up I have a vague understanding of projection matrices and looking at the functions provided by open3d I believe 2d > 3d projection can be done using the below code:

z = d / depth_scale

x = (u - cx) * z / fx

y = (v - cy) * z / fy

This works when depth is calculated from a solid plane. In my case depth is calculated from a central point (pinhole camera) and as such the projection extrudes away from the central point.

For example:

In image 1 the lengths of A,B,C may be 11m, 10m, 11m whilst in image 2 they will be all equal at 10m, 10m, 10m (these measurements are not accurate, just to prove the point).

The result is that using measurements from image 1, with the above formula, will give the perception that the object is curved (away from the center). How can I amend the formula to adjust for this?

If anyone has examples in Python that would be super useful.

Thank you

André Aichert André Aichert · Accepted Answer · 2020-02-03T12:50:34

1) Check what your definition of 'depth' is. It would appear your depth is measured as the distance to the principal plane, rather than the distance to the camera center. This would explain why your reconstructions look 'curved'

2) If you have read about projection matrices, then a good way to approach the problem is to use them. https://towardsdatascience.com/inverse-projection-transformation-c866ccedef1c

(Recommended book: Multiple view geometry, by Hartley & Zisserman)

How does 2d to 3d projection work when depth is taken from a central point?

1 Answers