I have a scene that I am rendering couple of cubes in it with openGL (program structure is not using GLUT,it is in win32 program structure but I just draw cubes with glutSolidCube
)
now I want to select these cubes by mouse by picking. this is what I am doing:
first when the user clicks the mouse button on the scene I get the mouse position and trying to find its coordinates in scene coordinates (templateSkeletons is a skeleton I created out of cubes, nothing more):
if (mouse.buttonPressed(Mouse::BUTTON_LEFT))
{
mouse.update();
templateSkeletons[0].selectionMode = true;
Vector3* points;
points = GetOGLPos();
templateSkeletons[0].setIntersectionPoints(points[0],points[1]);
}else
templateSkeletons[0].selectionMode = false;
this is the GerOGLPos
function that I am retrieving the coordinates in the scene (notice that I have my own camrea and its own projection matrix but I am fetching projection matrix here in this function just by calling glGetDoublev (GL_PROJECTION_MATRIX, projmatrix);
is this wrong and I should get my own camrea's projection matrix ? ) :
Vector3* GetOGLPos()
{Vector3 pointsOnLine[2];
double mvmatrix[16];
double projmatrix[16];
int viewport[4];
double dX, dY, dZ, dClickY,zz;
glGetIntegerv(GL_VIEWPORT, viewport);
glGetDoublev (GL_MODELVIEW_MATRIX, mvmatrix);
glGetDoublev (GL_PROJECTION_MATRIX, projmatrix);
dClickY = double (viewport[3] - mouse.yPos());
// OpenGL renders with (0,0) on bottom, mouse reports with (0,0) on top
//glReadPixels( mouse.xPos(), int(dClickY), 1, 1, GL_DEPTH_COMPONENT, GL_FLOAT, &zz );
gluUnProject ((double) mouse.xPos(), dClickY, 0.0, mvmatrix, projmatrix, viewport, &dX, &dY, &dZ);
pointsOnLine[0] = Vector3( (float) dX, (float) dY, (float) dZ );
gluUnProject ((double) mouse.xPos(), dClickY, 1.0, mvmatrix, projmatrix, viewport, &dX, &dY, &dZ);
pointsOnLine[1] = Vector3( (float) dX, (float) dY, (float) dZ );
return pointsOnLine;
}
now I suppose I have two points indicating my picking ray in the scene. now when I render the cubes I try to calculate the distance of the line created by the ray and the cube, and if it is smaller than a value I change the color of the cube to know that I picked it (jointsOfSkeleton indicates each cube that is created the skeleton nothing more, in here I am testing just the cube number 6 in array ):
if(selectionMode)
{
distToLine = Vector3::PointToLineDistance3D(rayPoints[0],rayPoints[1],Vector3::Vector3(jointsOfSkeleton[6].x,
jointsOfSkeleton[6].y,jointsOfSkeleton[6].z));
//distToLine = sqrt(distToLine);
if(distToLine < 0.5)
glColor3f(1.0,0.0,0.0);
else
glColor3f(1.0,1.0,1.0);
}
when I click on irrelevant positions on the window I see the colors of cube changes, it doesn't work right, I am watching the distances on the debugger and the distances doesn't look right. and this is function I used for finding Line-point distance:
static float PointToLineDistance3D(Vector3 a, Vector3 b, Vector3 point)
{
Vector3 lineDirection = b - a;
float t = (Vector3::dot(point,lineDirection) - Vector3::dot(lineDirection,a))/(Vector3::dot(lineDirection,lineDirection));
Vector3 direction;
direction.x = a.x + (lineDirection.x *t) - point.x;
direction.y = a.y + (lineDirection.y *t) - point.y;
direction.z = a.z + (lineDirection.z *t) - point.z;
float ShortestDistance = sqrtf((direction.x*direction.x)+(direction.y*direction.y)+(direction.z*direction.z));
return ShortestDistance;
}