Three.js/webgl RayCasting Sprites in Scene with Transparency or Alternatives

Question

I am working on a webgame which requires selection of drawn objects, so not in simple geometric shapes. To do this, I have been drawing them as sprites in a scene on a canvas using Three.JS. I have been racking my brain for months trying to figure out how to determine if the ray of the raycaster in my webgl context (using ThreeJS) is colliding with the transparent part of a sprite. I've searched and read as many posts as I could find, but only two solution posts I've found are:

var ctx =renderer.getContext("experimental-webgl", {preserveDrawingBuffer: true})
const pixel = new Uint8Array(4)
ctx.readPixels(10,30,1,1, ctx.RGBA,  ctx.UNSIGNED_BYTE,  pixel); log(pixel);

which requires using preserveDrawingBuffer, which would destroy my performance, or:

raycaster.intersectObject(scene.getObjectByName('some sprite')[0]].uv

and then using math wizardry to compare the UV coordinates with the image itself, I guess loaded into a buffer somewhere from scene.getObjectByName('some sprite').material.map.image.src.valueOf(), and then if the matching pixel or fuzzy area is transparent then we don't consider the sprite selected. This would be additionally difficult because the sprites are often rotated, scaled, center-offset, etc. I am not even 100% sure how I would implement that at all.

I am wondering if there is any other way, particularly a more 'proper' way to do this. Is there some way I can convert sprite images into/supply my images with some kind of meshes, then paint the mesh with the image precisely, so that the raytracer will actually have to use the correct object shape instead of just an image plane (sprite)? Would it be better to take every sprite and painstakingly model its edges in a 3d modeling program? I have tried every combination of "mesh," "geometry," "image," "sprite," etc and found no solution.

Any help is appreciated. Beggars can't be choosers, but I am also hoping to have animated objects eventually, so it would be additionally helpful if that was not rendered impossible by this solution.

Thank you

e: see @gman's comment below. I worked in the color picking system, only really changing emissive.setHex to material.color.set(id), then checked for color instead of emissives. works great, but seems to be slowing things down a bit. Will have to see if I implemented it badly or if I can make it more efficient. Thanks to you both!!

It's not clear to me what you're trying to do but maybe the techniques in this article would help. — gman
@gman I am working on attempting both of your solutions, as both are similar, but it will take me at least a day to implement either, have already spent 3 hours on it today. I will update this by Monday with any results. Note: i started with yours, and it'll take me a bit of time to basically rewrite my sprite creation/ rendering functions, as they are all dynamically populated. — egg peanut
as far as what I am trying to do, it is nearly the same as the article, just swap out randomly painted cubes with sprites, which are drawn, positioned, and scaled by what the canvas receives from the server. Hopefully it is acceptable that I wait until I've attempted to implement both until I choose the correct one, so I can update my post with the solution. — egg peanut
I could not implement your solution. The id returned is always zero on the pickhelper function, using console.log, it doesn't seem to be finding the object in the picking scene, which I checked and made sure is there. — egg peanut

YGilk1 YGilk1 · Accepted Answer · 2020-05-09T08:34:32

With an additional draw pass, you can render the scene into a texture with a dedicated material for every object, and pick the 2D point into it

You may create a basic material with a color matching every object's ID as described in this OpenGL hack (convert it to WebGL)

Use Scene.overrideMaterial to assign a material to all the objects and Object3D.onBeforeRender to set the color of each object before drawing them

It will have a cost, but I think it's worth a try

Three.js/webgl RayCasting Sprites in Scene with Transparency or Alternatives

1 Answers