ARKit Face Tracking SceneKit Object Moves Incorrectly

Question

I'm having trouble understanding SceneKit transforms and anchoring an object to a detected face. I have created a face detection app and have successfully applied masks, with and without texture. I also successfully applied "glasses" made from text ( "00" ) including an occlusion node.

In both cases, the objects move with the face as expected. However, when I create a simple hat made from two cylinders within ScendKit the behavior is totally unexpected.

First, I could not seem to anchor the hat to the face, but had to adjust the transforms which made the hat appear in a different place with almost every face. Even worse, the hat moves in the opposite direction to the face. Rotate the user face to the left, the hat moves to the right. Rotate the face up, the hat moves down.

Clearly, I'm missing something important here about anchoring objects to the face. Any guidance would be appreciated.

Xcode 10 beta 3, iOS 11.4.1 running on an iPhone X.

There is a separate class for hat, glasses, mask:

class Hat : SCNNode {

    init(geometry : ARSCNFaceGeometry) {

        geometry.firstMaterial?.colorBufferWriteMask = []
        super.init()
        self.geometry = geometry

        guard let url = Bundle.main.url(forResource: "hat", withExtension: "scn", subdirectory: "Models.scnassets") else {fatalError("missing hat resource")}
        let node = SCNReferenceNode(url: url)!
        node.load()

        addChildNode(node)

    }//init

    func update(withFaceAnchor anchor : ARFaceAnchor) {
        let faceGeometry = geometry as! ARSCNFaceGeometry
        faceGeometry.update(from: anchor.geometry)
    }//upadate


    required init?(coder aDecoder: NSCoder) {
        fatalError("(#function) has not been implemented")
    }//r init

}//class

A couple of the functions in the ViewController:

func renderer(_ renderer: SCNSceneRenderer, didUpdate node: SCNNode, for anchor: ARAnchor) {

    guard let faceAnchor = anchor as? ARFaceAnchor else {return}

    updateMessage(text: "Tracking your face")

    switch contentTypeSelected {
    case .none:
        break
    case .mask:
        mask?.update(withFaceAnchor: faceAnchor)
    case .glasses:
        glasses?.update(withFaceAnchor: faceAnchor)
    case .hat:
        hat?.update(withFaceAnchor: faceAnchor)
    }//switch

}//didUpdate

func createFaceGeometry() {
    updateMessage(text: "Creating face geometry")

    let device = sceneView.device!

    let maskGeometry = ARSCNFaceGeometry(device: device)!
    mask = Mask(geometry: maskGeometry, maskType : maskType)

    let glassesGeometry = ARSCNFaceGeometry(device: device)!
    glasses = Glasses(geometry: glassesGeometry)

    let hatGeometry = ARSCNFaceGeometry(device: device)!
    hat = Hat(geometry: hatGeometry)

}//createFaceGeometry

beyowulf beyowulf · Accepted Answer · 2018-08-09T01:32:15

The verisimilitude of a hat is going to depend on how well it can be position in relation to the face and how well it can appear situated in the scene (i.e. features that would be in front of the hat should occlude the hat itself). With that in mind, you'll want the face to occlude the hat. So your init for Hat should setup an occlusion node using the face geometry:

let occlusionNode: SCNNode

 init(geometry: ARSCNFaceGeometry) {

    /*
     Taken directly from Apple's sample code https://developer.apple.com/documentation/arkit/creating_face_based_ar_experiences
     */
    geometry.firstMaterial!.colorBufferWriteMask = []
    occlusionNode = SCNNode(geometry: geometry)
    occlusionNode.renderingOrder = -1

    super.init()

    addChildNode(occlusionNode)

    guard let url = Bundle.main.url(forResource: "hat", withExtension: "scn", subdirectory: "Models.scnassets") else {fatalError("missing hat resource")}
    let node = SCNReferenceNode(url: url)!
    node.load()

    addChildNode(node)
}

This will allow the face to appear in front of any virtual objects that have a z depth greater than the face mesh.

You will also want change let hatGeometry = ARSCNFaceGeometry(device: device)! to let hatGeometry = ARSCNFaceGeometry(device: device, fillMesh: true)! otherwise the hat will be visible through the eyes giving an uncanny, undesirable effect.

The next issue is to position the hat so that it appears believably in the scene.

Because we want the face to occlude a large part of the hat, it is best to position it in the y direct at the top of the face geometry. To do that successfully, you'll likely want your hat to have pivot point at the bottom center of the hat geometry and located at x = 0, y = 0 in your .scn file. For example the scene editor and node inspector might look something like:

Then in your func update(withFaceAnchor anchor : ARFaceAnchor) you can say

func update(withFaceAnchor anchor : ARFaceAnchor) {
    let faceGeometry = geometry as! ARSCNFaceGeometry
    faceGeometry.update(from: anchor.geometry)
    hat.position.y = faceGeometry.boundingSphere.radius
}

Finally for the z position of the hat you'll like want a slightly negative value as the bulk of a hat is behind one's face. -0.089 worked well for me.

ARKit Face Tracking SceneKit Object Moves Incorrectly

1 Answers