I would like to find a solution for taking a rotation represented as a matrix and then resetting one of it's components. Basically I want to be able to multiply a vector by this matrix and get a direction that is rotation around x and z axis and be constant along the y axis (up). I want to take object rotation and get the vector that represents gravity but in object local space and disregarding the yaw. So I want to reset the yaw.
I don't want to convert this to euler angles. I would prefer using a quaternion or doing some sequence of operations on the rotation matrix directly in order to avoid possible bugs with certain angles.
Ok, so I have the follwoing:
btTransform t;
mBody->getMotionState()->getWorldTransform(t);
btMatrix3x3 trans = t.getBasis().inverse();
btVector3 up = (trans * btVector3(0, 1, 0));
I realized that if I used quaternion then I got completely wrong results (why?). Now I'm getting a vector in object space that represents up vector in world space. BUT I want to rotate this vector so that it represents global up vector in object space WHEN MODEL HAS ZERO rotation around Y axis. So I have to somehow rotate this vector back. How?
