WebGL: how to prevent camera bounce when changing center of rotation

Question

I've hit a mental block of sorts, and was looking for some advice or suggestions. My problem is this:

I have a WebGL scene (I'm not using a 3rd party library, except gl-matrix), in which the user can rotate the camera up/down and left/right (rotate around X/Y axis). They can also rotate the model as well (yaw/pitch).

To see the problem, imagine the model has two blocks, A and B in the scene, with A at the center and B to the right (in the viewport), and the rotation center in the center of A. If the user rotates the model, it rotates about the center of block A. But if the user clicks on object B, I need to be able to change the center of rotation to B's center, but still maintain the current camera orientation. Currently, when the center of rotation switches to B, block B moves to the center of the screen, and block A moves to the left. Basically, the code always centers on the current center or rotation.

I use the following code for the modelview matrix update:

var mvMatrix = this.mvMatrix;

mat4.identity(mvMatrix);

mat4.translate(mvMatrix, mvMatrix, this.orbit);
mat4.rotateY(mvMatrix, mvMatrix, this.orbitYaw);
mat4.rotateX(mvMatrix, mvMatrix, this.orbitPitch);

mat4.translate(mvMatrix, mvMatrix, this.eye);
mat4.rotateY(mvMatrix, mvMatrix, this.eyeYaw);
mat4.rotateX(mvMatrix, mvMatrix, this.eyePitch);

I'm trying to figure out what the right yaw and pitch values for orbit and eye I should use in order to move back the current location and to achieve the present camera/eye orientation to avoid the "bounce" from one object to another as the rotation center moves.

I've searched a lot and can't seem to find how best to do this (my current attempt(s) have issues). Any sample code, or just good descriptions would be appreciated.

Edit

I followed gman's advice and tried the following code, but switching orbits just jumped around. My model is composed of multiple objects, and the orbit center can change, but after changing orbits, the orientation of the camera needs to remain steady, which is why I have to calculate the correction to the orbit yaw/pitch and eye yaw/pitch to put the eye back in the same spot and pointing in the same direction after changing orbits. BTW, I only have one orbit yaw and pitch, based on where the current orbit is, so that's a little different from gman's sample:

Camera.prototype.changeOrbit = function (newOrbit) {
var matA = mat4.create();
var matB = mat4.create();

mat4.translate(matA, matA, this.orbit);
mat4.rotateY(matA, matA, this.orbitYaw);
mat4.rotateX(matA, matA, this.orbitPitch);

mat4.translate(matB, matB, newOrbit);
mat4.rotateY(matB, matB, this.orbitYaw);
mat4.rotateX(matB, matB, this.orbitPitch);

var matInverseNewOrbit  = mat4.create();
var matNewOrbitToCamera = mat4.create();

mat4.invert(matInverseNewOrbit, matB);
mat4.multiply(matNewOrbitToCamera, matInverseNewOrbit, matA);

var m = matNewOrbitToCamera;

this.eye[0] = m[12];
this.eye[1] = m[13];
this.eye[2] = m[14];

this.eyePitch = ExtractPitch(m);
this.eyeYaw   = ExtractYaw(m);

this.update();
};

ExtractPitch and ExtractYaw work as gman had specified, but I do rotate around different axes since pitch is normally defined around the Y axis, and so on. Thanks for the suggestions, though.

Answer 1

I'm not sure I can explain this but basically:

When switching from A to B, at switch time,

1. Compute the matrix for the camera going around A (the code you have above). (camera)
2. Compute the matrix for B (matB)
3. Compute the inverse of the matrix for B. (inverseMatB)

4. Multiply camera by inverseMatB. (matBtoCamera)

   You now have a matrix that goes from B to the camera.

5. Decompose this matrix (matBToCamera) back into translation and rotation.

Unfortunately I don't know of a good decompose matrix function to point you at. I haven't needed one in a long time. Translation is basically elements 12, 13, 14 of your matrix. (Assuming you are using 16 element matrices which I think is what glMatrix uses).

var translation = [m[12], m[13], m[14]];

For rotation the upper/left 3x3 part of the matrix represents rotation. As long as there is no scaling or skewing involved, according to this page (http://nghiaho.com/?page_id=846) it's

var rotXInRadians = Math.atan2(m[9], m[10]);
var rotYInRadians = Math.atan2(-m[8], Math.sqrt(m[9] * m[9] + m[10] * m[10]));
var rotZInRadians = Math.atan2(m[4], m[0]);

Here's an example

http://jsfiddle.net/greggman/q7Bsy/

I'll paste the code here specific to glMatrix

// first let's make 3 nodes, 'a', 'b', and 'camera

var degToRad = function(v) {
  return v * Math.PI / 180;
}

var a = {
  name: "a",
  translation: [0, -50, -75],
  pitch: 0,
  yaw: degToRad(30),
};

var b = {
  name: "b",
  translation: [0, 100, 50],
  pitch: 0,
  yaw: degToRad(-75),
}

var camera = {
  name: "cam",
  translation: [0, 15, 10],
  pitch: 0,
  yaw: degToRad(16),
  parent: a,
};

Here's the code that computes the matrix of each

var matA = mat4.create();
mat4.identity(matA);
mat4.translate(matA, matA, a.translation);
mat4.rotateY(matA, matA, a.pitch);
mat4.rotateX(matA, matA, a.yaw);
a.mat = matA;

var matB = mat4.create();
mat4.identity(matB);
mat4.translate(matB, matB, b.translation);
mat4.rotateY(matB, matB, b.pitch);
mat4.rotateX(matB, matB, b.yaw);
b.mat = matB;

var matCamera = mat4.create();
mat4.identity(matCamera);

var parent = camera.parent;

mat4.translate(matCamera, matCamera, parent.translation);
mat4.rotateY(matCamera, matCamera, parent.pitch);
mat4.rotateX(matCamera, matCamera, parent.yaw);

mat4.translate(matCamera, matCamera, camera.translation);
mat4.rotateY(matCamera, matCamera, camera.pitch);
mat4.rotateX(matCamera, matCamera, camera.yaw);

camera.mat = matCamera;

and here's the code that swaps cameras

// Note: Assumes matrices on objects are updated.
var reparentObject = function(obj, newParent) {
  var matInverseNewParent = mat4.create();
  var matNewParentToObject = mat4.create();
  mat4.invert(matInverseNewParent, newParent.mat);
  mat4.multiply(matNewParentToObject, matInverseNewParent, obj.mat);

  var m = matNewParentToObject;
  obj.translation[0] = m[12];
  obj.translation[1] = m[13];
  obj.translation[2] = m[14];

  var rotXInRadians = Math.atan2(m[9], m[10]);
  var rotYInRadians = Math.atan2(-m[8], Math.sqrt(m[9] * m[9] + m[10] * m[10]));
  var rotZInRadians = Math.atan2(m[4], m[0]);

  obj.pitch = rotYInRadians;
  obj.yaw = rotXInRadians;
  obj.parent = newParent;
};

var newParent = camera.parent == a ? b : a;
reparentObject(camera, newParent);