'Ray' creation for raypicking not fully working

Question

I'm trying to implement a 'raypicker' for selecting objects within my project. I do not fully understand how to implement this, but I understand conceptually how it should work. I've been trying to learn how to do this, but most tutorials I find go way over my head. My current code is based on one of the recent tutorials I found, here.

After several hours of revisions, I believe the problem I'm having with my raypicker is actually the creation of the ray in the first place. If I substitute/hardcode my near/far planes with a coordinate that would undisputably be located within the region of a triangle, the picker identifies it correctly.

My problem is this: my ray creation doesn't seem to fully take my current "camera" or perspective into account, so camera rotation won't affect where my mouse is. I believe to remedy this I need something like using gluUnProject() or something, but whenever I used this the x,y,z coordinates returned would be incredibly small,

My current ray creation is a mess. I tried to use methods that others proposed initially, but it seemed like whatever method I tried it never worked with my picker/intersection function.

Here's the code for my ray creation:

void oglWidget::mousePressEvent(QMouseEvent *event)
{   

    QVector3D nearP = QVector3D(event->x()+camX, -event->y()-camY, -1.0);
    QVector3D farP = QVector3D(event->x()+camX, -event->y()-camY, 1.0);

    int i = -1;
    for (int x = 0; x < tileCount; x++)
    {
        bool rayInter = intersect(nearP, farP, tiles[x]->vertices);
        if (rayInter == true)
            i = x;
    }
    if (i != -1)
    {
        tiles[i]->showSelection();
    }
    else
    {
        for (int x = 0; x < tileCount; x++)
            tiles[x]->hideSelection();
    }
    //tiles[0]->showSelection();
}

To repeat, I used to load up the viewport, model & projection matrices, and unproject the mouse coordinates, but within a 1920x1080 window, all I get is values in the range of -2 to 2 for x y & z for each mouse event, which is why I'm trying this method, but this method doesn't work with camera rotation and zoom.

I don't want to do pixel color picking, because who knows I may need this technique later on, and I'd rather not give up after the amount of effort I put in so far

Answer 1

Take a look at

http://www.realtimerendering.com/intersections.html

Lot of help in determining intersections between various kinds of geometry

http://geomalgorithms.com/code.html also has some c++ functions one of them serves your purpose

Answer 2

As you seem to have problems constructing your rays, here's how I would do it. This has not been tested directly. You could do it like this, making sure that all vectors are in the same space. If you use multiple model matrices (or stacks thereof) the calculation needs to be repeated separately with each of them.

1. use pos = gluUnproject(winx, winy, near, ...) to get the position of the mouse coordinate on the near plane in model space; near being the value given to glFrustum() or gluPerspective()
2. origin of the ray is the camera position in model space: rayorig = inv(modelmat) * camera_in_worldspace
3. the direction of the ray is the normalized vector from the position from 1. to the ray origin: raydir = normalize(pos - rayorig)

On the website linked they use two points for the ray and they don't seem to normalize the ray direction vector, so this is optional.

Answer 3

Ok, so this is the beginning of my trail of breadcrumbs.

I was somehow having issues with the QT datatypes for the matrices, and the logic pertaining to matrix transformations.

This particular problem in this question resulted from not actually performing any transformations whatsoever.

Steps to solving this problem were:

• Converting mouse coordinates into NDC space (within the range of -1 to 1: x/screen width * 2 - 1, y - height / height * 2 - 1)
• grabbing the 4x4 matrix for my view matrix (can be the one used when rendering, or re calculated)
• In a new vector, have it equal the inverse view matrix multiplied by the inverse projection matrix.

In order to build the ray, I had to do the following:

• Take the previously calculated value for the matrices that were multiplied together. This will be multiplied by a vector 4 (array of 4 spots), where it will hold the previously calculated x and y coordinates, as well as -1, then +1.
• Then this vector will be divided by the last spot value of the entire vector
• Create another vector 4 which was just like the last, but instead of -1, put "1" .
• Once again divide that by its last spot value.

Now the coordinates for the ray have been created at the far and near planes, so it can intersect with anything along it in the scene.

I opened a series of questions (because of great uncertainty with my series of problems), so parts of my problem overlap in them too.

• In here, I learned that I needed to take the screen height into consideration for switching the origin of the y axis for a Cartesian system, since windows has the y axis start at the top left. Additionally, retrieval of matrices was redundant, but also wrong since they were never declared "properly".
• In here, I learned that unProject wasn't working because I was trying to pull the model and view matrices using OpenGL functions, but I never actually set them in the first place, because I built the matrices by hand. I solved that problem in 2 fold: I did the math manually, and I made all the matrices of the same data type (they were mixed data types earlier, leading to issues as well).
• And lastly, in here, I learned that my order of operations was slightly off (need to multiply matrices by a vector, not the reverse), that my near plane needs to be -1, not 0, and that the last value of the vector which would be multiplied with the matrices (value "w") needed to be 1.

Credits goes to those individuals who helped me solve these problems:

• srobins of facepunch, in this thread
• derhass from here, in this question, and this discussion