Object picking with Ray casting in elm-webgl

Question

Demo almost (?) working example: https://ellie-app.com/4h9F8FNcRPya1/1
For demo: Click to draw ray, and rotate camera with left and right to see ray. (As the origin is from the camera, you can't see it from the position it is created)

Context
I am working on an elm & elm-webgl project where I would like to know if the mouse is over an object when clicked. To do is I tried to implement a simple ray cast. What I need is two things:
1) The coordinate of the camera (This one is easy)
2) The coordinate/direction in 3D space of where was clicked

Problem
The steps to get from 2D view space to 3D world space as I understand are:
a) Make coordinates to be in a range of -1 to 1 relative to view port
b) Invert projection matrix and perspective matrix
c) Multiply projection and perspective matrix
d) Create Vector4 from normalised mouse coordinates
e) Multiply combined matrices with Vector4
f) Normalise result

Try so far
I have made a function to transform a Mouse.Position to a coordinate to draw a line to:

getClickPosition : Model -> Mouse.Position -> Vec3
getClickPosition model pos =
    let
        x =
            toFloat pos.x

        y =
            toFloat pos.y

        normalizedPosition =
            ( (x * 2) / 1000 - 1, (1 - y / 1000 * 2) )

        homogeneousClipCoordinates =
            Vec4.vec4
                (Tuple.first normalizedPosition)
                (Tuple.second normalizedPosition)
                -1
                1

        inversedProjectionMatrix =
            Maybe.withDefault Mat4.identity (Mat4.inverse (camera model))

        inversedPerspectiveMatrix =
            Maybe.withDefault Mat4.identity (Mat4.inverse perspective)

        inversedMatrix2 =
            Mat4.mul inversedProjectionMatrix inversedPerspectiveMatrix

        to =
            Vec4.vec4
                (Tuple.first normalizedPosition)
                (Tuple.second normalizedPosition)
                1
                1

        toInversed =
            mulVector inversedMatrix2 to

        toNorm =
            Vec4.normalize toInversed

        toVec3 =
            vec3 (Vec4.getX toNorm) (Vec4.getY toNorm) (Vec4.getZ toNorm)
    in
        toVec3

Result
The result of this function is that the rays are too much to the center to where I click. I added a screenshot where I clicked in all four of the top face of the cube. If I click on the center of the viewport the ray will be correctly positioned.

It feels close, but not quite there yet and I can't figure out what I am doing wrong!

I thought the function from Mat4.makeLookAt returns a multiplication of the projection and view matrix. package.elm-lang.org/packages/elm-community/linear-algebra/… I find no other reference of the projection matrix anywhere in elm-webgl — mahulst
Mat4.makeLookAt gives a view matrix, and Mat4.makePerspective gives a projection matrix. — Conrad Parker
Thanks, that does make sense and will help me to understand these things better — mahulst

mahulst mahulst · Accepted Answer · 2017-09-13T15:34:49

After trying other approaches I found a solution:

getClickPosition : Model -> Mouse.Position -> Vec3
getClickPosition model pos =
    let
        x =
            toFloat pos.x

        y =
            toFloat pos.y

        normalizedPosition =
            ( (x * 2) / 1000 - 1, (1 - y / 1000 * 2) )

        homogeneousClipCoordinates =
            Vec4.vec4
                (Tuple.first normalizedPosition)
                (Tuple.second normalizedPosition)
                -1
                1

        inversedViewMatrix =
            Maybe.withDefault Mat4.identity (Mat4.inverse (camera model))

        inversedProjectionMatrix =
            Maybe.withDefault Mat4.identity (Mat4.inverse perspective)

        vec4CameraCoordinates = mulVector inversedProjectionMatrix homogeneousClipCoordinates

        direction = Vec4.vec4 (Vec4.getX vec4CameraCoordinates) (Vec4.getY vec4CameraCoordinates) -1 0

        vec4WorldCoordinates = mulVector inversedViewMatrix direction

        vec3WorldCoordinates = vec3 (Vec4.getX vec4WorldCoordinates) (Vec4.getY vec4WorldCoordinates) (Vec4.getZ vec4WorldCoordinates)

        normalizedVec3WorldCoordinates = Vec3.normalize vec3WorldCoordinates

        origin = model.cameraPos

        scaledDirection = Vec3.scale  20 normalizedVec3WorldCoordinates

        destination = Vec3.add origin scaledDirection

    in
        destination

I left it as verbose as possible, if someone finds I use incorrect terminology please make a comment and I will update the answer.

I am sure there are lots of optimisations possible (Multiplying matrices before inverting or combining some of the steps.)

Updated the ellie app here: https://ellie-app.com/4hZ9s8S92PSa1/0

Object picking with Ray casting in elm-webgl

1 Answers