This is a very old question, but I just came across the exact same problem, as I was trying to adapt the camera code from learnopengl.com to the Arcball model...
Importantly, I didn't want to move to quaternions, as anything done with them can also be done using standard Linear Algebra.
A solution I found was to transform the arcball's coordinates into the camera coordinates (as if the arcball were rotating along with the camera).
I also rotate the Postion, Front and Up vectors of my camera by using the Rodrigues' formula, and then resort to glm:LookAt to return the final ViewMatrix (the one that is fed into the Vertex shader).
For the moment I have a quick and dirty solution, which looks as follows:
// Processes input received from a mouse input system.
void ProcessMouseMovement(glm::vec2 prevMouse, glm::vec2 curMouse)
{
glm::vec3 p1 = screen_to_arcball( curMouse );
glm::vec3 p2 = screen_to_arcball( prevMouse );
// Rotate arcball to camera coordinates
p1 = glm::vec3( glm::inverse(ViewMatrix) * glm::vec4(p1, 0.0));
p2 = glm::vec3( glm::inverse(ViewMatrix) * glm::vec4(p2, 0.0));
glm::vec3 axis = glm::cross(p1, p2);
float angle = std::acos(glm::dot(p1,p2));
if ( angle > 0.001f)
{
// Rotate
Position = rotate_axis_angle(Position, axis, angle);
Front = rotate_axis_angle(Front, axis, angle);
Up = rotate_axis_angle(Up, axis, angle);
ViewMatrix = glm::lookAt(Position, Position + Front, Up);
}
}
// Get Arcball coordinates from screen
glm::vec3 screen_to_arcball(const glm::vec2 &p)
{
const float dist = glm::dot(p, p);
if (dist < 0.999f)
{
// If we're on/in the sphere return the point on it
return glm::vec3(p.x, p.y, std::sqrt(1.f - dist));
}
else
{
// otherwise we project the point onto the sphere
const glm::vec2 proj = glm::normalize(p);
return glm::vec3(proj.x, proj.y, 0.f);
}
}
// Rotate vector given (angle,axis) using Rodrigues' formula
glm::vec3 rotate_axis_angle(glm:: vec3 &vec, glm:: vec3 axis, float angle)
{
axis = glm::normalize(axis);
glm::vec3 cross_axis = glm::cross(axis, vec);
vec = vec * std::cos(angle) + axis * glm::dot(axis,vec) * (1.0f - std::cos(angle)) + cross_axis * std::sin(angle);
return vec;
}
By the way, I took the screen_to_arcball function from a previous version of ArcballCamera.cpp in Ospray's repository. They have a quaternion-based implementation now, that looks similar to arcball_camera.cpp from Twinklebear.