How to integrate ARKit into GPUImage render with SCNRender?

Question

The graph is below:
ARFrame -> 3DModelFilter(SCNScene + SCNRender) -> OtherFilters -> GPUImageView.

Load 3D model:

NSError* error; 
SCNScene* scene =[SCNScene sceneWithURL:url options:nil error:&error];

Render 3D model:

SCNRenderer* render = [SCNRenderer rendererWithContext:context options:nil];
render.scene = scene;
[render renderAtTime:0];

Now，I am puzzle on how to apply ARFrame's camera transform to the SCNScene.

Some guess:

Can I assign ARFrame camera's transform to the transform of camera node in scene without any complex operation?
The ARFrame camera's projectMatrix do not have any help to me in this case?

update 2017-12-23.

First of all, thank @rickster for your reply. According to your suggestion, I add code in ARSession didUpdateFrame callback:

ARCamera* camera = frame.camera;
SCNMatrix4 cameraMatrix = SCNMatrix4FromMat4(camera.transform);
cameraNode.transform = cameraMatrix;

matrix_float4x4 mat4 = [camera projectionMatrixForOrientation:UIInterfaceOrientationPortrait viewportSize:CGSizeMake(375, 667) zNear:0.001 zFar:1000];
camera.projectionTransform = SCNMatrix4FromMat4(mat4);

Run app.
1. I can't see the whole ship, only part of it. So I add a a translation to the camera's tranform. I add the code below and can see the whole ship.

cameraMatrix = SCNMatrix4Mult(cameraMatrix, SCNMatrix4MakeTranslation(0, 0, 15));

2. When I move the iPhone up or down, the tracking seem's work. But when I move the iPhone left or right, the ship is follow my movement until disappear in screen.

I think there is some important thing I missed.

Did you ever get this working / matching ARSCNView? Im doing something similar and am also seeing some not quite correct results. Thanks! — vade

rickster rickster · Accepted Answer · 2017-12-20T20:01:14

ARCamera.transform tells you where the camera is in world space (and its orientation). You can assign this directly to the simdTransform property of the SCNNode holding your SCNCamera.

ARCamera.projectionMatrix tells you how the camera sees the world — essentially, what its field of view is. If you want content rendered by SceneKit to appear to inhabit the real world seen in the camera image, you'll need to set up SCNCamera with the information ARKit provides. Conveniently, you can bypass all the individual SCNCamera properties and set a projection matrix directly on the SCNCamera.projectionTransform property. Note that property is a SCNMatrix4, not a SIMD matrix_float4x4 as provided by ARKit, so you'll need to convert it:

scnCamera.projectionTransform = SCNMatrix4FromMat4(arCamera.projectionMatrix);

Note: Depending on how your view is set up, you may need to use ARCamera.projectionMatrixForOrientation:viewportSize:zNear:zFar: instead of ARCamera.projectionMatrix so you get a projection appropriate for your view's size and UI orientation.

How to integrate ARKit into GPUImage render with SCNRender?

1 Answers