Find Pitch and Yaw of Face Using Vision Framework

Question

Using VNFaceObservation to get the bounding box and landmark information about the face, but have not been able to find where to get the pitch and yaw of the face from the observation.

Have also tried getting pitch and yaw metadata from a CIDetector, but running both the CIDetector and the Vision Framework at the same time is CPU intensive.

    let metadataOutput = AVCaptureMetadataOutput()
    let metaQueue = DispatchQueue(label: "MetaDataSession")
    metadataOutput.setMetadataObjectsDelegate(self, queue: metaQueue)
    if captureSession.canAddOutput(metadataOutput) {
        captureSession.addOutput(metadataOutput)
    } else {
        print("Meta data output can not be added.")
    }


    let configurationOptions: [String: AnyObject] = [CIDetectorAccuracy: CIDetectorAccuracyHigh as AnyObject, CIDetectorTracking : true as AnyObject, CIDetectorNumberOfAngles: 11 as AnyObject]
    faceDetector = CIDetector(ofType: CIDetectorTypeFace, context: nil, options: configurationOptions)

Is there a way to use the VNFaceObservation data to find the pitch and yaw of the face?

VNFaceObservation provides Yaw and Roll as of iOS12, but annoyingly no pitch. — Ash

Ri_ Ri_ · Accepted Answer · 2018-04-26T14:21:49

As far as I am aware, measures for pitch and yaw aren't provided in CoreImage (I'm unfamiliar with Vision Framework).

It can however be detected using AVFoundation, as demonstrated in the sample project for this post

func metadataOutput(_ output: AVCaptureMetadataOutput, didOutput metadataObjects: [AVMetadataObject], from connection: AVCaptureConnection) {

    for metadataObject in metadataObjects as! [AVMetadataFaceObject] {
        DispatchQueue.main.async {
            self.avFaceID.text = "face ID: \(metadataObject.faceID)"
            self.avFaceRoll.text = "roll: \(Int(metadataObject.rollAngle))"
            self.avFaceYaw.text = "yaw: \(Int(metadataObject.yawAngle))"
        }
    }

}

Find Pitch and Yaw of Face Using Vision Framework

1 Answers