RealityKit – Image recognition and working with many scenes

Question

I've created an app using the RealityKit template file. Inside RealityComposer there are multiple scenes, all the scenes use image recognition that activates some animations.

Inside Xcode I have to load all the scenes as anchors and append those anchors to arView.scene.anchors array. The issue is an obvious one, as I present the physical 2D image one after the other I get multiple anchors piled on top of each other which is not desirable. I'm aware of arView.scene.anchors.removeAll() prior to loading the new anchor but my issue is this:

How do I check when a certain image has appeared to therefore remove the existing anchor and load the correct one? I've tried to look for something like there is in ARKit as didUpdate but I can't see anything similar in RealityKit.

Many thanks

Andy Fedoroff Andy Fedoroff · Accepted Answer · 2021-01-12T08:39:49

Foreword

RealityKit's AnchorEntity(.image) coming from RC, matches ARKit's ARImageTrackingConfig. When iOS device recognises a reference image, it creates Image Anchor (that conforms to ARTrackable protocol) that tethers a corresponding 3D model. And, as you understand, you must show just one reference image at a time (in your particular case AR app can't operate normally when you give it two or more images simultaneously).

Code snippet showing how if condition logic might look like:

import SwiftUI
import RealityKit

struct ContentView : View {
    var body: some View {            
        return ARViewContainer().edgesIgnoringSafeArea(.all)
    }
}

struct ARViewContainer: UIViewRepresentable {

    func makeUIView(context: Context) -> ARView {    
        let arView = ARView(frame: .zero)

        let id02Scene = try! Experience.loadID2()            
        print(id02Scene)     // prints scene hierarchy
        
        let anchor = id02Scene.children[0]
        print(anchor.components[AnchoringComponent] as Any)
        
        if anchor.components[AnchoringComponent] == AnchoringComponent(
                      .image(group: "Experience.reality", 
                              name: "assets/MainID_4b51de84.jpeg")) {
            
            arView.scene.anchors.removeAll()
            print("LOAD SCENE")
            arView.scene.anchors.append(id02Scene)
        }
        return arView
    }
 
    func updateUIView(_ uiView: ARView, context: Context) { }
}

ID2 scene hierarchy printed in console:

P.S.

You should implement SwiftUI Coordinator class (read about it here), and inside Coordinator use ARSessionDelegate's session(_:didUpdate:) instance method to update anchors properties at 60 fps.

Also you may use the following logic – if anchor of scene 1 is active or anchor of scene 3 is active, just delete all anchors from collection and load scene 2.

var arView = ARView(frame: .zero)

let id01Scene = try! Experience.loadID1()
let id02Scene = try! Experience.loadID2()
let id03Scene = try! Experience.loadID3()

func makeUIView(context: Context) -> ARView {    
    arView.session.delegate = context.coordinator

    arView.scene.anchors.append(id01Scene)
    arView.scene.anchors.append(id02Scene)
    arView.scene.anchors.append(id03Scene)
    return arView
}

...

func session(_ session: ARSession, didUpdate frame: ARFrame) {
    if arView.scene.anchors[0].isActive || arView.scene.anchors[2].isActive {
        arView.scene.anchors.removeAll()
        arView.scene.anchors.append(id02Scene)
        print("Load Scene Two")
    }
}

RealityKit – Image recognition and working with many scenes

1 Answers