webgl and glsl picking

Question

Im implementing webgl picking from scratch, and have decided to go down the GLSL route, as oppose to ray intersection testing.

so Im rendering the whole scene into a separate frame buffer, assigning each object a unique color which is passed to the fragment shader as a uniform variable. when the scene is rendered then I gl.readPixels() the buffer, and get the color values at the click coordinates (I invert the coord system to account for GL differing from the browsers coord system).

the issue Im having is that shaders represent colors passed to the gl_FragColor shader output as vec4 floats with 0.0-1.0 range per color channel, whereas gl.readPixels() returns color channels as integers in the 0-255 range... doing this translation loses some precision, and might create picking errors (if there are a lot of objects in the scene (more then 255), or if the integer-to-float rounding is larger then the granularity needed to differentiate between different object ID's).

does anyone have an idea how to resolve this, or point me in the right direction? can readPixels() return float values for color channels in the 0.0-1.0 range? can I pack a single object ID spread over multiple channels (so that Im not limited to a single channel and only being able to pick 255 objects?)

thank you for your help

Answer 1

There exists some tricks to read floats from GPU to CPU in WebGL:

http://lab.concord.org/experiments/webgl-gpgpu/webgl.html

More specific:

• Method 1: based on the Carlos Scheidegger's idea. See here for more information.
• Method 2: based on the Ultraist's idea. See here for more information.

Answer 2

This is an old question, but just to offer...

Can I pack a single object ID spread over multiple channels (so that Im not limited to a single channel and only being able to pick 255 objects)?

You can reliably encode 8 bits per color component retrieved by 8-bit gl.readPixels. In your fragment shader, you could use float redComponent = float(intObjectId) / 255.0. If you put that in gl_FragColor, it will reliably return as the number it started as, if it was in 0 to 255. No rounding is needed.

You can encode a larger integer by mod and division, like

    int objectId = 12345; // up to 65535...
    float f = float(objectId);
    // put low 8 bits into .r, and next 8 bits into .g
    gl_FragColor = vec4(mod(f,256.) / 255., mod(floor(f / 256.0), 256.0) / 255.0, 0,1);

The thing to know is that the ranges map from floating [0.0, 1.0] to int [0,255]. And they map right to the middle of the range, so 0.0 up to 0.5/255 maps to 0, 0.5/255 to 1.5/255 maps to 1, and so on. Very stable!

Answer 3

The OpenGL specification states precisely how conversions from int to float and from float to int should be done.

See section 2.3.5, "Fixed-Point Data Conversions":

• Conversion from integer type to float should be done by dividing by 2^b-1, i.e. 255.0 in the case of bytes, as in david van brink's answer.
• Conversion from a floating-point type to an integer type by the GPU is always done by rounding to the nearest integer. As long as the internal framebuffer format has enough precision, this should be OK: 8 bit unsigned integer, 16 bit float (which has a 10 bit mantissa), and 32 bit float all work for that purpose.

Answer 4

Unfortunately, while you can use the OES_texture_float extension to render to a floating point frame buffer texture, it looks like you still can't read them as floats using readPixels (see https://bugzilla.mozilla.org/show_bug.cgi?id=681903 and https://groups.google.com/forum/#!topic/webgl-dev-list/s83_IvYbPM4)

Looks like it's a future feature, but doesn't look like it's implemented anywhere yet.

If you're avoiding the ray intersection test because it's too costly, it may be best to use an optimized structure for your objects, e.g. bounding volumes or octtrees/kd-trees.

Answer 5

I know that in OpenGL you can create textures from pointers (I mean, getting the piece of the memory that pointer is pointing to as texture's content). I believe that opposite process in possible also, but not quite sure how, and could it be done. Something like:

*p = myTexture;     // texture -> pointer
my_data = p[i][j];  // read byte data

If that could be done, than you could read content as byte data and then convert it to float (or whatever suits you). But if is that even possible and how exactly could be done in JavaScript, that question remains... Still, this the way I would go.

Hope this helps.