SceneKit matrix transformation to match camera angle

Question

I'm building a UIPanGestureRecognizer so I can move nodes in 3D space.

Currently, I have something that works, but only when the camera is exactly perpendicular to the plane, my UIPanGestureRecognizer looks like this:

@objc func handlePan(_ sender:UIPanGestureRecognizer) {
  let projectedOrigin = self.sceneView!.projectPoint(SCNVector3Zero)

  let viewCenter = CGPoint(
    x: self.view!.bounds.midX,
    y: self.view!.bounds.midY
  )

  let touchlocation = sender.translation(in: self.view!)

  let moveLoc = CGPoint(
    x: CGFloat(touchlocation.x + viewCenter.x),
    y: CGFloat(touchlocation.y + viewCenter.y)
  )

  let touchVector = SCNVector3(x: Float(moveLoc.x), y: Float(moveLoc.y), z: Float(projectedOrigin.z))
  let worldPoint = self.sceneView!.unprojectPoint(touchVector)
  let loc = SCNVector3( x: worldPoint.x, y: 0, z: worldPoint.z )

  worldHandle?.position = loc
}

The problem happens when the camera is rotated, and the coordinates are effected by the perspective change. Here is you can see the touch position drifting:

Related SO post for which I used to get to this position: How to use iOS (Swift) SceneKit SCNSceneRenderer unprojectPoint properly

It referenced these great slides: http://www.terathon.com/gdc07_lengyel.pdf

If the grid is a plane (else you can add a plane and make it invisible) you can simply do a hittest on the plane node to get the coordinates on the plane. Works for the tap gesture but also very accurately for the pan gesture (e.g. hittest when the pan starts and hittest again everytime the gesture state changed is active). This will exclude the camera factor as well as the orientation of the plane. You know the number of tiles and their width/height so you can simply divide the coordinates by the width of a tile to get the column, similar for a row. — Xartec
That works however the coordinates of the SCNVector3 on the plane are affected by the perspective distortion and the hits are detected in the wrong spot, works perfectly when the plane is perpendicular however. — JP Silvashy
No, the localCoordinates (of the hitresult) on the plane will always be consistent regardless of camera settings. You can then convert the resulting position to worldspace to get the location for nodes you want to place on that position. — Xartec
Maybe my issue is with translating the position, I've edited my question with a screen recording of the touch position drifting. — JP Silvashy
Thanks, that clears things up a bit :) As Rickster mention in the post you referenced to that approach only works if the plane is perpendicular to the camera. That is why I suggest using a hittest as it takes the camera and plane orientation out of the equation. I will post an answer with sample code shortly. — Xartec

Xartec Xartec · Accepted Answer · 2018-01-28T18:25:31

The tricky part of going from 2D touch position to 3D space is obviously the z-coordinate. Instead of trying to convert the touch position to an imaginary 3D space, map the 2D touch to a 2D plane in that 3D space using a hittest. Especially when movement is required only in two direction, for example like chess pieces on a board, this approach works very well. Regardless of the orientation of the plane and the camera settings (as long as the camera doesn't look at the plane from the side obviously) this will map the touch position to a 3D position directly under the finger of the touch and follow consistently.

I modified the Game template from Xcode with an example. https://github.com/Xartec/PrecisePan/

The main parts are:

the pan gesture code:

// retrieve the SCNView
    let scnView = self.view as! SCNView
    // check what nodes are tapped
    let p = gestureRecognize.location(in: scnView)
    let hitResults = scnView.hitTest(p, options: [SCNHitTestOption.searchMode: 1, SCNHitTestOption.ignoreHiddenNodes: false])

    if hitResults.count > 0 {
        // check if the XZPlane is in the hitresults
        for result in hitResults {
            if result.node.name == "XZPlane" {
                //NSLog("Local Coordinates on XZPlane %f, %f, %f", result.localCoordinates.x, result.localCoordinates.y, result.localCoordinates.z)

                //NSLog("World Coordinates on XZPlane %f, %f, %f", result.worldCoordinates.x, result.worldCoordinates.y, result.worldCoordinates.z)
                ship.position = result.worldCoordinates
                ship.position.y += 1.5
                return;
            }
        }
    }

The addition of a XZ plane node in viewDidload:

let XZPlaneGeo = SCNPlane(width: 100, height: 100)
let XZPlaneNode = SCNNode(geometry: XZPlaneGeo)
XZPlaneNode.geometry?.firstMaterial?.diffuse.contents = UIImage(named: "grid")
XZPlaneNode.name = "XZPlane"
XZPlaneNode.rotation = SCNVector4(-1, 0, 0, Float.pi / 2)
//XZPlaneNode.isHidden = true
scene.rootNode.addChildNode(XZPlaneNode)

Uncomment the isHidden line to hide the helper plane and it will still work. The plane obviously needs to be large enough to fill the screen or at least the portion where the user is allowed to pan.

By setting a global var to hold a startWorldPosition of the pan (in state .began) and comparing it to the hit worldPosition in the state .change you can determine the delta/translation in world space and translate other objects accordingly.

SceneKit matrix transformation to match camera angle

1 Answers