I have designed a C++ class that abstracts the user from trackball rotation, zooming and panning. I have got the rotation (using trackball) and zooming working as expected. However, panning does not behave as expected. When I pick a point and drag, I expect that on finishing drag, the picked point continues to be under the mouse. My understanding of panning in perspective projection is as follows. The target and the camera position, both will be affected by a pan operation. The camera target and the camera position (eye) should be translated proportional to the drag. The proportionality (may not be constant) should be based on z depth.
Panning is straight forward in orthographic projection but poses a problem in perspective. It will be useful if one can explain the math and the implementation detail for OpenGL.