OpenGL world coordinates from mouse coordinates

Question

I'm using the Haskell code below to get the OpenGL world coordinates from the mouse coordinates (the code is a bit long but this is a minimal code to draw a cube). My expectation is that the function worldFromScreen given at the end returns the world coordinates from the mouse coordinates (given in pixels from the top-left corner). Thanks to the keyboard-mouse callback I defined, these supposed world coordinates are printed in the terminal when the user right-clicks the mouse.

The code generates a cube of length 1 in front view:

The cube has length 1 and is centered at the origin, so the (x,y) coordinates of the vertices of this face should be (+/-0.5,+/-0.5). However when I right-click on a vertex of this face, I get coordinates (+/-0.2,+/-0.2) approximately. What is this 0.2 ? Is it possible to get (+/-0.5,+/-0.5) by changing something in my code ?

module Basic
  where
import           Graphics.Rendering.OpenGL
import           Graphics.UI.GLUT

white,black,red :: Color4 GLfloat
white      = Color4    1    1    1    1
black      = Color4    0    0    0    1
red        = Color4    1    0    0    1

display :: DisplayCallback
display = do
  clear [ColorBuffer, DepthBuffer]
  loadIdentity
  preservingMatrix $ do
    materialDiffuse Front $= red
    renderObject Solid $ Cube 1
  swapBuffers

resize :: Size -> IO ()
resize s@(Size w h) = do
  viewport $= (Position 0 0, s)
  matrixMode $= Projection
  loadIdentity
  perspective 45.0 (w'/h') 1.0 100.0
  lookAt (Vertex3 0 0 (-3)) (Vertex3 0 0 0) (Vector3 0 1 0)
  matrixMode $= Modelview 0
  where
    w' = realToFrac w
    h' = realToFrac h

keyboardMouse :: KeyboardMouseCallback
keyboardMouse key state _ position@(Position x y) =
  case (key,state) of
    (MouseButton LeftButton, Down) -> print (x,y)
    (MouseButton RightButton, Down) -> do
      (sx, sy, sz) <- worldFromScreen position
      print (sx, sy, sz)
    _                              -> return ()

main :: IO ()
main = do
  _ <- getArgsAndInitialize
  _ <- createWindow ""
  windowSize $= Size 500 500
  initialDisplayMode $= [RGBAMode, DoubleBuffered, WithDepthBuffer]
  clearColor $= black
  materialAmbient FrontAndBack $= black
  lighting $= Enabled
  light (Light 0) $= Enabled
  position (Light 0) $= Vertex4 200 200 (-500) 1
  ambient (Light 0) $= white
  diffuse (Light 0) $= white
  specular (Light 0) $= white
  depthFunc $= Just Less
  shadeModel $= Smooth
  displayCallback $= display
  reshapeCallback $= Just resize
  keyboardMouseCallback $= Just keyboardMouse
  idleCallback $= Nothing
  mainLoop

worldFromScreen :: Position -> IO (GLdouble, GLdouble, GLdouble)
worldFromScreen (Position sx sy) = do
  viewport@(_, Size _ viewSizeY) <- get viewport
  projectionMatrix <- get (matrix $ Just Projection) :: IO (GLmatrix GLdouble)
  modelviewMatrix <- get (matrix $ Just $ Modelview 0) :: IO (GLmatrix GLdouble)
  let screenPos = Vertex3 (fromIntegral sx) (fromIntegral ((viewSizeY - 1) - sy)) 0
  (Vertex3 wx wy wz) <- unProject screenPos projectionMatrix modelviewMatrix viewport
  return (wx, wy, wz)

Screen coordinates are 2D coordinates. To get the 3D world coordinates you need the third coordinate. — Ripi2
Yes Ripi, but I want only the (x,y) coordinates. It looks like the x or y world coordinate I get is 0.5/(h-0.5) where h is the z coordinate of the camera position. I observed that for different values of h. — Stéphane Laurent
If the porjection is orthogonal then any z-value may suffix. With perspective projection this is not true. Draw a fustrum, join points with the camera and see where they intersect the near plane. — Ripi2
Thank you @Ripi2, I begin to understand. I will try with an orthographic projection (I don't know what it is yet, but I suppose it is an orthogonal projection? I will see later I'm busy now). — Stéphane Laurent

Harish Harish · Accepted Answer · 2018-06-21T04:32:00

I don't know Haskell, so I won't be able to provide the working code, but I think I understand your problem.

You have a 2D screen position that you unProject into 3D and get a point. This point is a position on your camera's near plane in world space. That is what your "0.2" is. Now you haven't specified the z value but I can compute this to be around -1.8.

When you click on corners you get (+/- 0.2, +/- 0.2) for a cube whose length in world is 1 so that corner should be (+/- 0.5, +/- 0.5).

Note that what you get is a point on camera's near plane in world space, it is NOT a point on your cube.

This point can be used along with your camera position to generate a ray and then intersect this ray with a plane to obtain the real position on that plane.

From your code,

lookAt (Vertex3 0 0 (-3)) (Vertex3 0 0 0) (Vector3 0 1 0)

The camera is at (0,0,-3).

If we consider the top right corner of your cube, lets say the 3D point you obtain is (0.2, 0.2, 1.2). Using these two points, a Ray can be constructed with the camera as the origin and the 3D point (0.2, 0.2, -1.8) as the second point.

Ray( (0.0, 0.0, -3.0), (0.2, 0.2, -1.8) )

This ray can then intersect the plane where your cube is to obtain the point you are looking for. As your cube is centered at the origin and you are looking straight at it, the plane can be defined by normal (0,0,1) and the origin (0,0,0).

Plane(Normal (0.0, 0.0, 1.0), Point (0.0, 0.0, 0.0)).

Now if we intersect the Ray with the Plane, we get :

(0.5, 0.5, 0.0)

Presumably the point you are looking for.

PS : I used this link "Plane line intersection" to quickly test the ray/plane intersection.

Hope this helps.

OpenGL world coordinates from mouse coordinates

1 Answers