arkit anchor or node visible in camera and sitting to left or right of frustum

6
votes

How can i detect if an ARAnchor is currently visible in the camera, i need to test when the camera view changes. I want to put arrows on the edge of the screen that point in the direction of the anchor when not on screen. I need to know if the node sits to the left or right of the frustum.

I am now doing this but it says pin is visible when it is not and the X values seem not right? Maybe the renderer frustum does not match the screen camera?

 var deltaTime = TimeInterval()

 public func renderer(_ renderer: SCNSceneRenderer, updateAtTime time: TimeInterval) {
        deltaTime = time - lastUpdateTime
        if deltaTime>1{

            if let annotation = annotationsByNode.first {

                let node = annotation.key.childNodes[0]

                if !renderer.isNode(node, insideFrustumOf: renderer.pointOfView!)
                {
                    print("Pin is not visible");
                }else {
                    print("Pin is visible");
                }
                let pnt = renderer.projectPoint(node.position)

                print("pos ", pnt.x, " ", renderer.pointOfView!.position)


            }
            lastUpdateTime = time
        }

    }

Update: The code works to show if node is visible or not, how can i tell which direction left or right a node is in relation to the camera frustum?

update2! as suggested answer from Bhanu Birani

let screenWidth = UIScreen.main.bounds.width
let screenHeight = UIScreen.main.bounds.height
let leftPoint = CGPoint(x: 0, y: screenHeight/2)
let rightPoint = CGPoint(x: screenWidth,y: screenHeight/2)

let leftWorldPos = renderer.unprojectPoint(SCNVector3(leftPoint.x,leftPoint.y,0))
let rightWorldPos = renderer.unprojectPoint(SCNVector3(rightPoint.x,rightPoint.y,0))
let distanceLeft = node.position - leftWorldPos
let distanceRight = node.position - rightWorldPos
let dir = (isVisible) ? "visible" : ( (distanceLeft.x<distanceRight.x) ? "left" : "right")

I got it working finally which uses the idea from Bhanu Birani of the left and right of the screen but i get the world position differently, unProjectPoint and also get a scalar value of distance which i compare to get the left/right direction. Maybe there is a better way of doing it but it worked for me

public func renderer(_ renderer: SCNSceneRenderer, updateAtTime time: TimeInterval) {
        deltaTime = time - lastUpdateTime
        if deltaTime>0.25{

            if let annotation = annotationsByNode.first {
                guard let pointOfView = renderer.pointOfView else {return}
                let node = annotation.key.childNodes[0]
                let isVisible = renderer.isNode(node, insideFrustumOf: pointOfView)

                let screenWidth = UIScreen.main.bounds.width
                let screenHeight = UIScreen.main.bounds.height
                let leftPoint = CGPoint(x: 0, y: screenHeight/2)
                let rightPoint = CGPoint(x: screenWidth,y: screenHeight/2)

                let leftWorldPos = renderer.unprojectPoint(SCNVector3(leftPoint.x, leftPoint.y,0))
                let rightWorldPos =  renderer.unprojectPoint(SCNVector3(rightPoint.x, rightPoint.y,0))
                let distanceLeft = node.worldPosition.distance(vector: leftWorldPos)
                let distanceRight = node.worldPosition.distance(vector: rightWorldPos)

                //let pnt = renderer.projectPoint(node.worldPosition)
                //guard let pnt = renderer.pointOfView!.convertPosition(node.position, to: nil) else {return}

                let dir = (isVisible) ? "visible" : ( (distanceLeft<distanceRight) ? "left" : "right")
                print("dir" , dir, " ", leftWorldPos , " ", rightWorldPos)
                lastDir=dir
                delegate?.nodePosition?(node:node, pos: dir)
            }else {
               delegate?.nodePosition?(node:nil, pos: lastDir )
            }
            lastUpdateTime = time
        }

extension SCNVector3
{

    /**
     * Returns the length (magnitude) of the vector described by the SCNVector3
     */
    func length() -> Float {
        return sqrtf(x*x + y*y + z*z)
    }

    /**
     * Calculates the distance between two SCNVector3. Pythagoras!
     */
    func distance(vector: SCNVector3) -> Float {
        return (self - vector).length()
    }


}
2
swiftscenekitarkit
Look into SceneKit constraints — you can have one node that stays permanently “attached” to the camera, but that rotates to point toward another node regardless of whether that node is onscreen. – rickster
@rickster what i was looking in doing is putting an arrow on either left or right of the screen that points to where the node is when its off camera, when its on camera i dont need an arrow, its basically a location map pin, so that you know which way to rotate the camera to get it on screen – tsukimi
In that case, you might look into coordinate space conversion methods on SCNNode. Convert an object of interest’s position into camera coordinates, then look at the X value — negative is to the left of the camera, positive to the right. Your frustum test already tells you whether something is off camera, and the position conversion gets you rough direction. – rickster
@rickster if you could provide a little sample code as an answer it would help, i am from unity background and new to scenekit. The frustum test i am using now doesn't seem to work it always prints visible – tsukimi
Haven’t had the time to work up some code, so the comments are here to get you started. Now that you’ve clarified intent I can come back with an answer once I do... – rickster

2 Answers

12
votes

Project the ray from the from the following screen positions:

  • leftPoint = CGPoint(0, screenHeight/2) (centre left of the screen)
  • rightPoint = CGPoint(screenWidth, screenHeight/2) (centre right of the screen)

Convert CGPoint to world position:

  • leftWorldPos = convertCGPointToWorldPosition(leftPoint)
  • rightWorldPos = convertCGPointToWorldPosition(rightPoint)

Calculate the distance of node from both world position:

  • distanceLeft = node.position - leftWorldPos
  • distanceRight = node.position - rightWorldPos

Compare distance to find the shortest distance to the node. Use the shortest distance vector to position direction arrow for object.

Here is the code from tsukimi to check if the object is in right side of screen or on left side:

public func renderer(_ renderer: SCNSceneRenderer, updateAtTime time: TimeInterval) {
        deltaTime = time - lastUpdateTime
        if deltaTime>0.25{

            if let annotation = annotationsByNode.first {
                guard let pointOfView = renderer.pointOfView else {return}
                let node = annotation.key.childNodes[0]
                let isVisible = renderer.isNode(node, insideFrustumOf: pointOfView)

                let screenWidth = UIScreen.main.bounds.width
                let screenHeight = UIScreen.main.bounds.height
                let leftPoint = CGPoint(x: 0, y: screenHeight/2)
                let rightPoint = CGPoint(x: screenWidth,y: screenHeight/2)

                let leftWorldPos = renderer.unprojectPoint(SCNVector3(leftPoint.x, leftPoint.y,0))
                let rightWorldPos =  renderer.unprojectPoint(SCNVector3(rightPoint.x, rightPoint.y,0))
                let distanceLeft = node.worldPosition.distance(vector: leftWorldPos)
                let distanceRight = node.worldPosition.distance(vector: rightWorldPos)

                //let pnt = renderer.projectPoint(node.worldPosition)
                //guard let pnt = renderer.pointOfView!.convertPosition(node.position, to: nil) else {return}

                let dir = (isVisible) ? "visible" : ( (distanceLeft<distanceRight) ? "left" : "right")
                print("dir" , dir, " ", leftWorldPos , " ", rightWorldPos)
                lastDir=dir
                delegate?.nodePosition?(node:node, pos: dir)
            }else {
               delegate?.nodePosition?(node:nil, pos: lastDir )
            }
            lastUpdateTime = time
        }

Following is the class to help performing operations on vector

extension SCNVector3 {

    init(_ vec: vector_float3) {
        self.x = vec.x
        self.y = vec.y
        self.z = vec.z
    }

    func length() -> Float {
        return sqrtf(x * x + y * y + z * z)
    }

    mutating func setLength(_ length: Float) {
        self.normalize()
        self *= length
    }

    mutating func setMaximumLength(_ maxLength: Float) {
        if self.length() <= maxLength {
            return
        } else {
            self.normalize()
            self *= maxLength
        }
    }

    mutating func normalize() {
        self = self.normalized()
    }

    func normalized() -> SCNVector3 {
        if self.length() == 0 {
            return self
        }

        return self / self.length()
    }

    static func positionFromTransform(_ transform: matrix_float4x4) -> SCNVector3 {
        return SCNVector3Make(transform.columns.3.x, transform.columns.3.y, transform.columns.3.z)
    }

    func friendlyString() -> String {
        return "(\(String(format: "%.2f", x)), \(String(format: "%.2f", y)), \(String(format: "%.2f", z)))"
    }

    func dot(_ vec: SCNVector3) -> Float {
        return (self.x * vec.x) + (self.y * vec.y) + (self.z * vec.z)
    }

    func cross(_ vec: SCNVector3) -> SCNVector3 {
        return SCNVector3(self.y * vec.z - self.z * vec.y, self.z * vec.x - self.x * vec.z, self.x * vec.y - self.y * vec.x)
    }
}




extension SCNVector3{
    func distance(receiver:SCNVector3) -> Float{
        let xd = receiver.x - self.x
        let yd = receiver.y - self.y
        let zd = receiver.z - self.z
        let distance = Float(sqrt(xd * xd + yd * yd + zd * zd))

        if (distance < 0){
            return (distance * -1)
        } else {
            return (distance)
        }
    }
}

Here is the code snippet to convert tap location or any CGPoint to world transform.

@objc func handleTap(_ sender: UITapGestureRecognizer) {

    // Take the screen space tap coordinates and pass them to the hitTest method on the ARSCNView instance
    let tapPoint = sender.location(in: sceneView)
    let result = sceneView.hitTest(tapPoint, types: ARHitTestResult.ResultType.existingPlaneUsingExtent)

    // If the intersection ray passes through any plane geometry they will be returned, with the planes
    // ordered by distance from the camera
    if (result.count > 0) {

        // If there are multiple hits, just pick the closest plane
        if let hitResult = result.first {
            let finalPosition = SCNVector3Make(hitResult.worldTransform.columns.3.x + insertionXOffset,
                                                       hitResult.worldTransform.columns.3.y + insertionYOffset,
                                                       hitResult.worldTransform.columns.3.z + insertionZOffset
                    );
        }
    }
}

Following is the code to get hit test results when there's no plane found.

// check what nodes are tapped
let p = gestureRecognize.location(in: scnView)
let hitResults = scnView.hitTest(p, options: [:])
// check that we clicked on at least one object
if hitResults.count > 0 {
    // retrieved the first clicked object
    let result = hitResults[0] 
}
0
votes

This answer is a bit late but can be useful for someone needing to know where a node is in camera space relatively to the center (e.g. top left corner, centered ...).

You can get your node position in camera space using scene.rootNode.convertPosition(node.position, to: pointOfView).

In camera space,

  • (isVisible && (x=0, y=0)) means that your node is in front of the camera.

  • (isVisible && (x=0.1)) means that the node is a little bit on the right.

Some sample code :

public func renderer(_ renderer: SCNSceneRenderer, updateAtTime time: TimeInterval) {
    deltaTime = time - lastUpdateTime
    if deltaTime>0.25{

        if let annotation = annotationsByNode.first {
            guard let pointOfView = renderer.pointOfView else {return}
            let node = annotation.key.childNodes[0]
            let isVisible = renderer.isNode(node, insideFrustumOf: pointOfView)

            // Translate node to camera space
            let nodeInCameraSpace = scene.rootNode.convertPosition(node.position, to: pointOfView)
            let isCentered = isVisible && (nodeInCameraSpace.x < 0.1) && (nodeInCameraSpace.y < 0.1)
            let isOnTheRight = isVisible && (nodeInCameraSpace.x > 0.1)
            // ...

            delegate?.nodePosition?(node:node, pos: dir)
        }else {
           delegate?.nodePosition?(node:nil, pos: lastDir )
        }
        lastUpdateTime = time
    }