Runtime problem on Apple ARKit Face Recognition project (ARKitFaceExample)

Question

Apple has released this:

https://developer.apple.com/documentation/arkit/creating_face-based_ar_experiences

a while ago. I tried it and it produces a runtime problem.

I'm using an iPhone XR and the project is building successfully and is working normally. But at least one process should be executed in the main thread, but I'm not able to figure out, which part this is :/

The error Message looks like

2019-04-26 20:17:40.360763+0200 ARKitFaceExample[16979:3438178] [DYMTLInitPlatform] platform initialization successful
2019-04-26 20:17:40.551196+0200 ARKitFaceExample[16979:3438133] Metal GPU Frame Capture Enabled
2019-04-26 20:17:40.551498+0200 ARKitFaceExample[16979:3438133] Metal API Validation Enabled
2019-04-26 20:17:40.710761+0200 ARKitFaceExample[16979:3438133] [MC] System group container for systemgroup.com.apple.configurationprofiles path is /private/var/containers/Shared/SystemGroup/systemgroup.com.apple.configurationprofiles
2019-04-26 20:17:40.711381+0200 ARKitFaceExample[16979:3438133] [MC] Reading from public effective user settings.
=================================================================
Main Thread Checker: UI API called on a background thread: -[UIApplication applicationState]
PID: 16979, TID: 3438188, Thread name: com.apple.CoreMotion.MotionThread, Queue name: com.apple.root.default-qos.overcommit, QoS: 0
Backtrace:
4   libobjc.A.dylib                     0x000000021789f6f4 <redacted> + 56
5   CoreMotion                          0x000000021dfe9638 CoreMotion + 292408
6   CoreMotion                          0x000000021dfe9b68 CoreMotion + 293736
7   CoreMotion                          0x000000021dfe9a78 CoreMotion + 293496
8   CoreMotion                          0x000000021e0178a8 CoreMotion + 481448
9   CoreMotion                          0x000000021e0178ec CoreMotion + 481516
10  CoreFoundation                      0x000000021862b78c <redacted> + 28
11  CoreFoundation                      0x000000021862b074 <redacted> + 276
12  CoreFoundation                      0x0000000218626368 <redacted> + 2276
13  CoreFoundation                      0x0000000218625764 CFRunLoopRunSpecific + 452
14  CoreFoundation                      0x0000000218626498 CFRunLoopRun + 84
15  CoreMotion                          0x000000021e017280 CoreMotion + 479872
16  libsystem_pthread.dylib             0x00000002182a5920 <redacted> + 132
17  libsystem_pthread.dylib             0x00000002182a587c _pthread_start + 48
18  libsystem_pthread.dylib             0x00000002182addcc thread_start + 4
2019-04-26 20:17:40.745827+0200 ARKitFaceExample[16979:3438188] [reports] Main Thread Checker: UI API called on a background thread: -[UIApplication applicationState]
PID: 16979, TID: 3438188, Thread name: com.apple.CoreMotion.MotionThread, Queue name: com.apple.root.default-qos.overcommit, QoS: 0
Backtrace:
4   libobjc.A.dylib                     0x000000021789f6f4 <redacted> + 56
5   CoreMotion                          0x000000021dfe9638 CoreMotion + 292408
6   CoreMotion                          0x000000021dfe9b68 CoreMotion + 293736
7   CoreMotion                          0x000000021dfe9a78 CoreMotion + 293496
8   CoreMotion                          0x000000021e0178a8 CoreMotion + 481448
9   CoreMotion                          0x000000021e0178ec CoreMotion + 481516
10  CoreFoundation                      0x000000021862b78c <redacted> + 28
11  CoreFoundation                      0x000000021862b074 <redacted> + 276
12  CoreFoundation                      0x0000000218626368 <redacted> + 2276
13  CoreFoundation                      0x0000000218625764 CFRunLoopRunSpecific + 452
14  CoreFoundation                      0x0000000218626498 CFRunLoopRun + 84
15  CoreMotion                          0x000000021e017280 CoreMotion + 479872
16  libsystem_pthread.dylib             0x00000002182a5920 <redacted> + 132
17  libsystem_pthread.dylib             0x00000002182a587c _pthread_start + 48
18  libsystem_pthread.dylib             0x00000002182addcc thread_start + 4
2019-04-26 20:17:52.404466+0200 ARKitFaceExample[16979:3438187] [SceneKit] Error: Scene <SCNScene: 0x283b7a260> is modified within a rendering callback of another scene (<SCNScene: 0x283b68000>). This is not allowed and may lead to crash

with the following message on runtime:

I hope someone can help me :/

Answer 1

The same as there: ARKit template Xcode project Main Thread Checker log console

Because it is my answer, I will also paste it here.

(Of course you can disable checker, but sometimes it can be helpful. Check https://stackoverflow.com/a/45689250/7183675 for this.)

Error is inside Apple framework, so only way I found do work around it was subclassing UIApplication class and check application state on main thread this way:

1) Add main.swift file (this name is really important!) with these lines

import UIKit

    UIApplicationMain(
        CommandLine.argc,
        CommandLine.unsafeArgv,
        NSStringFromClass(MyApplicationClass.self),
        NSStringFromClass(MyDelegateClass.self)
    )

2) Remove @UIApplicationMain from MyDelegateClass if it was there or project wont compile because of multiple entry points

3) In MyApplicationClass.swift add this:

    import UIKit

    class MyApplicationClass: UIApplication {
    let semaphore = DispatchSemaphore(value: 0)
    override var applicationState: UIApplication.State {
        if Thread.current == Thread.main {
            return super.applicationState
        }

        var toReturn =  UIApplication.State.inactive

        DispatchQueue.main.async { [weak self] in
            guard let self = self else {return}
            toReturn = self.superAppState()
            self.semaphore.signal()
        }
        semaphore.wait()
        return toReturn
    }

    private func superAppState() -> UIApplication.State {
        return super.applicationState
    }
}

Now you are calling UIApplication.applicationState on main thread, so issue won't occur.

Well, there is one more issue. If CMMotionManager is initialized from main thread then if will block main thread and wait for UIApplication.State. That means deadlock. In this case you can't set your semaphore. Only way to return credible state avoiding deadlock would be implementing application state this way:

4)

import UIKit

        class MyApplicationClass: UIApplication {

    private static var configured = false

    private var state = UIApplication.State.inactive

    override var applicationState: UIApplication.State {
        if !MyApplicationClass.configured {
            MyApplicationClass.configured = true
            NotificationCenter.default.addObserver(self, selector: #selector(setStatus(_:)), name: UIApplication.didBecomeActiveNotification, object: nil)
            NotificationCenter.default.addObserver(self, selector: #selector(setStatus(_:)), name: UIApplication.willResignActiveNotification, object: nil)
            NotificationCenter.default.addObserver(self, selector: #selector(setStatus(_:)), name: UIApplication.willEnterForegroundNotification, object: nil)
            NotificationCenter.default.addObserver(self, selector: #selector(setStatus(_:)), name: UIApplication.didEnterBackgroundNotification, object: nil)
        }
        if Thread.current == Thread.main {
            return super.applicationState
        }
        return state
    }

    @objc func setStatus(_ notif: Notification) {
        switch notif.name {
        case UIApplication.didBecomeActiveNotification:
            state = .active
        case UIApplication.willResignActiveNotification:
            state = .inactive
        case UIApplication.willEnterForegroundNotification:
            state = .background
        case UIApplication.didEnterBackgroundNotification:
            state = .background
        default:
            state = .inactive
        }
    }
}

Personally I don't like last solution, but I wasn't able to find another workaround

Answer 2

In ViewController.swift file, inside an extension of View Controller and rendering() method add a DispatchQueue.main.async method.

DispatchQueue manages the execution of work items. Each work item submitted to a queue is processed on a pool of threads managed by the system.

extension ViewController: ARSCNViewDelegate {

    func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) {

        DispatchQueue.main.async {

            guard let faceAnchor = anchor as? ARFaceAnchor else { return }
            self.currentFaceAnchor = faceAnchor

            if node.childNodes.isEmpty, let contentNode = self.selectedContentController.renderer(renderer, nodeFor: faceAnchor) {
                node.addChildNode(contentNode)
            }
        }
    }
}

Also, convert your project's Swift 4.2 to Swift 5.0 code, and use iOS 12.2 target.

And possibly, this block of code might be more useful than the previous one:

extension ViewController: ARSCNViewDelegate {

    func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) {

        DispatchQueue.global().async {

            guard let faceAnchor = anchor as? ARFaceAnchor else { return }
            self.currentFaceAnchor = faceAnchor

            DispatchQueue.main.async {

                if node.childNodes.isEmpty, let contentNode = self.selectedContentController.renderer(renderer, nodeFor: faceAnchor) {
                    node.addChildNode(contentNode)
                }
            }
        }
    }
}

Here's my Debug Area's messages: