Picking triangles in OpenGL core profile when using glDrawElements

Question

I am drawing a mesh of triangles using glDrawElements and would like to be able to pick/select a triangle using the mouse click. The mesh of triangles can be very big.

In fixed-function OpenGL there is the possibility of using GL_SELECT: http://content.gpwiki.org/index.php/OpenGL:Tutorials:Picking .. However I am only interested in using OpenGL core profile.

Another possibility would be to use 'color coding':

http://www.lighthouse3d.com/opengl/picking/index.php?color1

http://www.opengl.org/resources/faq/technical/selection.htm

.. but as far as I know its not possible to indicate per triangle information yet when using glDrawElements?

Finally I could do CPU based picking by shooting a pick ray through the mouse location, but this would be quite slow since I guess I would have to transform the triangles on the CPU, so I would prefer a GPU based solution.

Does anyone have a suggestion on what is the best way do picking when using glDrawElements in OpenGL core profile?

Answer 1

About the 'color coding' approach, you could use gl_PrimitiveID to fill the color coded buffer(s) using a proper fragment shader, this would basically give you the index of the drawn triangle.

About the CPU based picking, you could use an existing library that handles the acceleration structures, and ray-mesh intersection, such as Bullet or Opcode.

The 'best' option for you depends of your use case, hard to tell.

Answer 2

Try seeing it from this perspective: OpenGL is only meant for drawing things. Picking should be done in another way. If you insist on using OpenGL for it (not unreasonable), use a FBO with a integer 16 bit single channel color buffer attachment, into which you render the object ID. Picking reduces to read the one pixel for the ID.

Finally I could do CPU based picking by shooting a pick ray through the mouse location, but this would be quite slow since I guess I would have to transform the triangles on the CPU, so I would prefer a GPU based solution.

I recommend this. However instead of transforming all the triangles, just inversely transform your one picking ray. You should manage you scene in some spatial subdivision structure anyway, so testing against that should be trivial.

Answer 3

Well you can indeed provide per triangle information on a DrawElements call. You then will have to provide the corresponding arrays (the former COLOR_ARRAY, TEXTURE_COORD_ARRAY, etc., now commonly called VertexAttribArrays). This would be probably the easiest solution if you insist on the GPU.

However, usually the way to go would be to fire a ray from the click point into your scene. Consequently, what you will have to do is not to transform each triangle, but to calculate an intersection test between your ray and your triangles. If the ray intersects, you hit it, if not you didn't. This indeed could come at quite some cost if you do it brute force.

However, usually you will have your triangles contained within some spatial data structure (i.e. your scene graph will/can/should have a few representations) like a octree (http://en.wikipedia.org/wiki/Octree) for example. This will allow for a collision response in quite a few tests. Additionally you can take into account the the final size a triangle will occupy on the screen (so it would be quite meaningless to select subpixel triangles in most cases).

And if you really want to get fancy, you can put the whole process to the GPU, too (one example that sounds interesting is e.g. http://blog.xeolabs.com/ray-picking-in-scenejs, but there will be more coming up when you use google).

I personally thing the easiest to implement is solution b) Software Ray Picking, then a) and finally GPU picking. Start of with the easiest, improve if you figure it is not fast enough. Don't be caught by too much premature optimization (e.g. ray-cast picking works perfectly well on my iPhone ;). Have fun and good luck with your picking algo.