ARKit set ARAnchor transform based on touch location

Question

I'm playing around with the AR starter app on XCode 9 where anchors are created in a scene on tap:

override func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?) {
  guard let sceneView = self.view as? ARSKView else {
    return
  }

  // Create anchor using the camera’s current position
  if let currentFrame = sceneView.session.currentFrame {

    // Create a transform with a translation of 0.2 meters in front     
    // of the camera
    var translation = matrix_identity_float4x4
    translation.columns.3.z = -0.2
    let transform = simd_mul(currentFrame.camera.transform, translation)

    // Add a new anchor to the session
    let anchor = ARAnchor(transform: transform)
    sceneView.session.add(anchor: anchor)
  }
}

This always results in the anchor being created in the middle of the screen regardless of where I tap, which makes sense, as we're getting the current camera transform and applying only a translation in the z axis to it.

I would like the anchor instead to be placed where I actually tapped with my finger. I can get the location of the tap using touches.first?.location(in: sceneView), i.e., just the distance from the top left corner of the screen in points, but I'm unsure how to apply these 2D pt coordinates to the anchor transform in meters, nor which axis they apply to.

Answer 1

I assume you're referring to Apple's ARKitExample project "Placing Virtual Objects in Augmented Reality".

Have a look at the method VirtualObject.translateBasedOnScreenPos(_:instantly:infinitePlane:) that is being called when you're moving an (already placed) model in the screen—that basically needs to solve the same problems that you describe.

You'll find that this in turn calls ViewController.worldPositionFromScreenPosition(_:objectPos:infinitePlane:).

Extracted from this method, their approach is:

1. Always do a hit test against exisiting plane anchors first. (If any such anchors exist & only within their extents.)

2. Collect more information about the environment by hit testing against the feature point cloud, but do not return the result yet.

3. If desired or necessary (no good feature hit test result): Hit test against an infinite, horizontal plane (ignoring the real world).

4. If available, return the result of the hit test against high quality features if the hit tests against infinite planes were skipped or no infinite plane was hit.

5. As a last resort, perform a second, unfiltered hit test against features. If there are no features in the scene, the result returned here will be nil.

As you can see, they consider various aspects that may-or-may-not apply to your use-case. Consider studying and re-using (parts of) their approach.