ARKit: Reproducing the Project Point function

4
votes

I'm attempting to reproduce the ARCamera's project point function, but for some reason the values are not matching up properly. I am taking the ARCamera's projection matrix and view matrix and applying basic CG perspective transform math, (PV) * p, but the NDC values do not match the pixel values given from the ARCamera's project point function. Any ideas? Am I forgetting something?

Some more detail:

Basically, I'm trying to take an ARFrame a the click of a button, and then trying to replicate the functionality of https://developer.apple.com/documentation/arkit/arcamera/2923538-projectpoint. I'm attempting to do this with https://developer.apple.com/documentation/arkit/arcamera/2887458-projectionmatrix and https://developer.apple.com/documentation/arkit/arcamera/2921672-viewmatrix, making sure all of the inputs match for both parts. CG size is used to transform the coordinates from NDC space to image space.

EDIT: Solution found, check comments below.

1
scenekitarkit
There's not enough detail in your question to say. But one thing to look into might be the difference between the camera image's aspect ratio and that of the view. - rickster
I've added some more detail regarding the documentation of what I'm using. Checking the aspect ratio is a good idea, but I'd be surprised if the internal projectionMatrix of the ARCamera is different than what's used in the projectPoint function. - John DeCorato
Have you made any progress at this? Is the internal projectionMatrix of the ARCamera different than what's used in the projectPoint function? - Mozahler
Check out this link. I used this library for my puzzle app "Nosie" in the app store. You can learn a ton from playing with it. github.com/ebay/HeadGazeLib - Mozahler
tbh I got into computers so they could do my math for me! My recommendation: Connect your phone via cable to your mac. Build the simplest possible test example and run under the debugger with breakpoints anywhere you're not 100% sure. Change the code, rerun and understand what changed and why. - Mozahler

1 Answers

1
votes

The problem turned out to be projection_matrix sometimes does not correctly find the device orientation. The correct approach is to use projectionMatrix(for:viewportSize:zNear:zFar:).