How to calculate the camera position from Vuforia GL matrix?

Question

I calculate the camera position of a SCNScene that is rendered in Vuforia. However the object is not staying fixed on the marker but jumping around when moving. The cube in the scene appears only orthographically, no matter how the device is moved around the sides cannot be seen.

The camera position is calculated with every frame:

// Get model view matrix
Vuforia::Matrix44F modelViewMatrix = Vuforia::Tool::convertPose2GLMatrix(result->getPose());

// Convert to extrinsic matrix
SCNMatrix4 extrinsic = [self SCNMatrix4FromVuforiaMatrix44: modelViewMatrix];
SCNMatrix4 inverted = SCNMatrix4Invert(extrinsic);

// Set new position of SCNCamera
cameraNode.transform = inverted;

The camera projection matrix is calculated when the camera is stared:

// Get device camera calibration    
const Vuforia::CameraCalibration& cameraCalibration = Vuforia::CameraDevice::getInstance().getCameraCalibration();
projectionGLMatrix = Vuforia::Tool::getProjectionGL(cameraCalibration, 2.0f, 5000.0f);

// Convert matrix
GLKMatrix4 glkMatrix;

for(int i=0; i<16; i++) {
    glkMatrix.m[i] = projectionGLMatrix.data[i];
}

// Convert matrix
SCNMatrix4 projectionTransform = SCNMatrix4FromGLKMatrix4(glkMatrix);
cameraNode.camera.projectionTransform = projectionTransform;

What am I doing wrong here?

Update 1

The projection matrix is now calculated like this, when the camera starts:

const Vuforia::CameraCalibration& cameraCalibration = Vuforia::CameraDevice::getInstance().getCameraCalibration();
Vuforia::Matrix44F vuforiaMatrix = Vuforia::Tool::getProjectionGL(cameraCalibration, 2.0f, 5000.0f);
matrix_float4x4 simdMatrix = simdMatrixWithVuforiaMatrix44F(vuforiaMatrix);
cameraNode.camera.projectionTransform = SCNMatrix4FromMat4(simdMatrix);

The camera position is updated with every frame:

Vuforia::Matrix44F modelViewMatrix = Vuforia::Tool::convertPose2GLMatrix(result->getPose());
matrix_float4x4 simdMatrix = simdMatrixWithVuforiaMatrix44F(modelViewMatrix);
cameraNode.transform = SCNMatrix4FromMat4(simdMatrix);

NSLog(@"camera position: x%lf, y%lf, z%lf, rotation: x%lf, y%lf, z%lf", _cameraNode.position.x, _cameraNode.position.y, _cameraNode.position.z, _cameraNode.rotation.x, _cameraNode.rotation.y, _cameraNode.rotation.z);

By moving the device and observing the logging of the camera position (left graph) and rotation (right graph) it seems that the axes are:

The axes for rotation are different from position axes. Also a rotation around the remaining axis (untitled in the graph below) does not have any influence on the cameraNode.rotation.x value which floats around 0.999.

What is wrong here?

Cameron Lowell Palmer Cameron Lowell Palmer · Accepted Answer · 2017-02-04T06:29:28

You're experiencing classic - "I did something wrong with the matrix" behaviour that takes lots of trial-and-error to resolve. I also think you've been reading stuff on Vuforia's website which is almost never helpful. :)

Re-arranging the matrices is probably where you're going wrong. The SCNMatrix is supposed to be directly compatible with OpenGL. I actually put everything into the simd::matrix4x4 and use the built in SCNMatrix4FromMat4 to convert them to SCNMatrix.

SceneKit uses matrices to represent coordinate space transformations, which in turn can represent the combined position, rotation or orientation, and scale of an object in three-dimensional space. SceneKit matrix structures are in row-major order, so they are suitable for passing to shader programs or OpenGL APIs that accept matrix parameters.

So, to sum up... I believe you should delete:

SCNMatrix4 inverted = SCNMatrix4Invert(extrinsic);

As for the copying of the GLMatrix to an SCNMatrix, without trying it, should be correct. Compare the result of the projection matrix generated using your method to mine. They should be identical.

Projection matrix

I'm getting the projection matrix like this:

const Vuforia::Matrix44F projectionMatrix = Vuforia::Tool::getProjectionGL(cameraCalibration, nearPlane, farPlane);
simdMatrixWithVuforiaMatrix44F(projectionMatrix);

I convert the matrix to a simd::matrix4x4 (I spend a lot of time in C++-land) which is just an Apple defined struct for which SceneKit has support.

#include <simd/simd.h>

matrix_float4x4 simdMatrixWithVuforiaMatrix44F(const Vuforia::Matrix44F &matrix)
{
    vector_float4 col0 = { matrix.data[0], matrix.data[1], matrix.data[2], matrix.data[3] };
    vector_float4 col1 = { matrix.data[4], matrix.data[5], matrix.data[6], matrix.data[7] };
    vector_float4 col2 = { matrix.data[8], matrix.data[9], matrix.data[10], matrix.data[11] };
    vector_float4 col3 = { matrix.data[12], matrix.data[13], matrix.data[14], matrix.data[15] };

    return matrix_from_columns(col0, col1, col2, col3);
}

Back in the view controller

let extrinsic = SCNMatrix4FromMat4(projectionMatrix)
_cameraNode?.camera?.projectionTransform = extrinsic

Framemarker poses

I have an object called a Framemarker that actually contains both the identifier and pose, but that pose is just the same simd::matrix4x4 as the projection matrix.

for framemarker in framemarkers {
        switch framemarker.identifier {
        case 337:
            let pose = SCNMatrix4FromMat4(framemarker.pose)
            _firstNode?.transform = pose
            break
        case 357:
            let pose = SCNMatrix4FromMat4(framemarker.pose)
            _secondNode?.transform = pose
            break
        default:
            break
        }
}

How to calculate the camera position from Vuforia GL matrix?

Update 1

1 Answers

Projection matrix

Framemarker poses