Sampling perspective (sub)views from fisheye image

Question

I have a problem trying to select virtual subimages (lets call them ROIs) in a fisheye image which then I want to undistort.

I have the fisheye calibration parameters following. Scaramuzza calibration model. What I would like to do is creating an array of virtual pinhole cameras (53° FOV) with a degree of overlap among them and sampling their correspondent subimage from the fisheye's. So imagine that my fisheye FOV is aproximately 180° and I want to cover that whole FOV with my 53° virtual cameras with 20° overlap among them, i.e, I would need roughly 180/(53-20)=5.45 cameras (I am not 100% sure about this calculation have to think a bit more about it). Correct me if I am wrong: is it the same to undistort the whole fisheye image and extracting the pinhole views from there than inferring the ROIs in the fisheye domain and undistort each of them separately? If not, I guess the second is the right way to go but how should I proceed. How can I know which area from the fisheye camera belong to each 53°FOV camera? My intuition tells me that the ones pointing towards the edge of the fisheye should have a more distorted ROI than the ones near the center.

I am not really good at understanding projective space and how to operate in there. Hope you guys can help me anyhow.

ElPotac ElPotac · Accepted Answer · 2020-12-16T10:55:55

Sorry but the title was misleading as I didnt know how to face the problem. What I meant with ROIs were perspective views from the fisheye view. They can also be interpreted as tangential views to the hemisphere that the fisheye produces in camera/world coordinates. I got a working solution.

First, you have to define a virtual perspective camera model with its intrinsic and extrinsic parameters, in particular we want a fov of 53ª and we want the perspective views to densely sample the whole fisheye domain, therefore we will need to place several at different angles wrt to the fisheye camera along the x axis and y axis. The extrinisics are given wrt to the fisheye pose. In this case, I only care about rotated virtual cameras with a [0,0,0] translation. The size parameter depicts the perspective image size.

def getVirtualCameraMatrix(viewfield, size, x_angle=0, y_angle=0, z_angle=0):

   f = 0.5*size[1] / np.tan(np.radians(0.5*viewfield))

   intrinsic_matrix = np.array([[f, 0, size[1]/2],
                             [0, f, size[0]/2],
                             [0, 0, 1]])

   rmat_x = np.array([[1,             0,                      0],
                   [0, np.cos(np.radians(x_angle)), -np.sin(np.radians(x_angle))],
                   [0, np.sin(np.radians(x_angle)), np.cos(np.radians(x_angle))]])

   rmat_y = np.array([[np.cos(np.radians(y_angle)), 0, np.sin(np.radians(y_angle))],
                   [0,                           1,                 0],
                   [-np.sin(np.radians(y_angle)), 0, np.cos(np.radians(y_angle))]])

   rmat_z = np.array([[np.cos(np.radians(z_angle)), -np.sin(np.radians(z_angle)), 0],
                   [np.sin(np.radians(z_angle)), np.cos(np.radians(z_angle)),  0],
                   [0,             0,                      1]])

   params = {'rotation': rmat_z @ rmat_y @ rmat_x,
          'translation': np.array([0, 0, 0]),
          'intrinsics': {
              'image_size': size,
              'focal_length': f,
              'principal_point': [size[0]/2, size[1]/2],
              'matrix': intrinsic_matrix}}

   return params

Then, to get each perspective view we must go from target to source, i.e, from the perspective view to fisheye to fetch the pixel values in the latter. Should proceed as follows:

Build a pixel coordinates meshgrid
Backproject each pixel coordinate to camera coords as:

K^-1 * pixel_coordinates

Where K^-1 ins the inverse of params['intrinsics']['matrix']. (you can get it with np.linalg.inv)

Transform the 3D points in the perspective camera coordinates to the fisheye camera coordinates applying the inverse rotation to step 2, i.e:

E^-1 * K^-1 * pixel_coordinates

where E^-1 is the inverse of params['rotation']

Project 3 to the fisheye image to get the tangential/perspective view using scaramuzza world2cam method. The link is in the question post

Sampling perspective (sub)views from fisheye image

1 Answers