Raycasting with InstancedMesh, InstancedBufferGeometry, custom shader

Question

Basic, I can't get raycasting to work with them. My guess is my matrix coordinate calculation method is wrong. Don't know how to do it right.

I set vertex position and offset in vertexShader, and in InstancedMesh, I set the same offset, expecting the the raycast can get the an instanceID, but nothing intersects. You can find my entire code here.

I tried to adapt an official raycasting example here, but can't figure out where I did wrong. My hodgepodge uses: InstancedMesh, InstancedBufferGeometry, custom shader together. My objective is to learn how it works.

My question is where I did wrong?

My vertex shader:

    precision highp float;

    uniform mat4 modelViewMatrix;
    uniform mat4 projectionMatrix;


    attribute vec3 position;
    attribute vec4 color;
    attribute vec3 offset;

    varying vec3 vPosition;
    varying vec4 vColor;



    void main() {
        vColor = vec4(color);
        vPosition = offset*1.0 + position;
        gl_Position = projectionMatrix * modelViewMatrix * vec4( vPosition, 1.0 );
        // if gl_Position not set, nothing is shown

    }

My InstancedMesh matrix setting:

        for(let i = 0; i < SQUARE_COUNT; i++)  {

            transform.position.set(offsets[i], offsets[i+1], offsets[i+2] )
            transform.updateMatrix()


            mesh.setMatrixAt(i, transform.matrix)
        }

The offsets is set before as following:

        for(let i = 0; i < SQUARE_COUNT; i++ ) {

            offsets.push( 0 + i*0.05, 0 + i*0.05, 0 + i*0.05); // same is set in InstancedMesh
            colors.push( Math.random(), Math.random(), Math.random(), Math.random() );
        }

manthrax manthrax · Accepted Answer · 2020-02-08T22:08:18

The raycaster has no awareness of any nonstandard transformation that you do in your vertex shader. It's just the way it works. It has no way of knowing that you are doing: vPosition = offset*1.0 + position; in your shader.

It works by assuming that you are running the bog standard vertex shader with no additional transforms. It assumes that every object you are casting against has a well defined/computed bounding box as well.

If you are going to use raycasting, you may have to make a non-rendered scene that represents your objects in their final rendered positions, and cast against that.

Raycasting with InstancedMesh, InstancedBufferGeometry, custom shader

1 Answers