Using SwiftUI and RealityKit for AR Experiences

Augmented Reality (AR) is revolutionizing how users interact with the digital world by blending virtual elements with real-world environments. Creating compelling AR experiences requires robust frameworks, and for Apple ecosystem development, SwiftUI and RealityKit provide a powerful combination. This blog post explores how to use SwiftUI with RealityKit to build immersive AR applications.

Understanding SwiftUI and RealityKit

• SwiftUI: Apple’s declarative UI framework enables developers to create user interfaces using a simple and readable syntax. SwiftUI automatically handles UI updates based on state changes, simplifying UI development.
• RealityKit: Apple’s 3D rendering and AR engine provides realistic rendering, animation, physics, and spatial audio capabilities. RealityKit integrates seamlessly with ARKit to understand the real world, enabling developers to place virtual content into AR scenes.

Why Combine SwiftUI and RealityKit?

• Modern Development: Leverages Apple’s latest technologies for efficient and future-proof AR development.
• Simplified UI: Uses SwiftUI for creating user interface elements and RealityKit for AR rendering.
• Enhanced AR Experiences: Delivers advanced AR features such as realistic rendering and physics simulations.

Setting Up Your AR Project

Step 1: Create a New Xcode Project

Launch Xcode and create a new project. Select the “Augmented Reality App” template under the iOS tab.

Step 2: Configure Project Settings

Ensure that your project includes the necessary configurations:

• Set the “Deployment Target” to iOS 13 or later to support RealityKit.
• Verify that the Info.plist file contains the NSCameraUsageDescription key with a description of why your app needs camera access.

Step 3: Project Structure Overview

Your project structure will include:

• ContentView.swift: The main SwiftUI view.
• Scene.rcproject: The RealityKit scene file where you can design your AR environment using Reality Composer.

Integrating RealityKit with SwiftUI

Step 1: Create a ARViewContainer

Since RealityKit is a UIKit-based framework, you’ll need to create a SwiftUI View that hosts an ARView. Here’s how:

import SwiftUI
import RealityKit

struct ARViewContainer: UIViewRepresentable {
    func makeUIView(context: Context) -> ARView {
        let arView = ARView(frame: .zero)

        // Load the "Box" scene from the "Experience" Reality File
        let boxAnchor = try! Experience.loadBox()

        // Add the box anchor to the scene
        arView.scene.anchors.append(boxAnchor)

        return arView
    }

    func updateUIView(_ uiView: ARView, context: Context) {}
}

Explanation:

• ARViewContainer conforms to UIViewRepresentable, allowing SwiftUI to manage a UIView instance.
• makeUIView creates an ARView and loads a predefined scene named “Box” from the “Experience” Reality File.
• updateUIView is left empty as no dynamic updates are required in this example.

Step 2: Integrate ARViewContainer into SwiftUI View

Use ARViewContainer within your SwiftUI view:

import SwiftUI
import RealityKit

struct ContentView : View {
    var body: some View {
        ARViewContainer().edgesIgnoringSafeArea(.all)
    }
}

#if DEBUG
struct ContentView_Previews : PreviewProvider {
    static var previews: some View {
        ContentView()
    }
}
#endif

Step 3: Load and Display an AR Scene

You can either load a scene created with Reality Composer or programmatically generate the scene.

Here’s an example of loading a scene created in Reality Composer:

1. Create a .rcproject File:

Open Reality Composer and create a new project. Add a 3D object to the scene. You can drag and drop a .usdz or .scn file into the scene.

2. Create an Anchor:

Add an anchor to the scene by clicking the “+” button and selecting an appropriate anchor type (e.g., “Plane Anchor”). Attach your 3D object to this anchor.

3. Save and Add to Xcode:

Save the Reality Composer project (e.g., Scene.rcproject) and drag it into your Xcode project.

The previous code example already showed how to load the scene in ARViewContainer:

let boxAnchor = try! Experience.loadBox()
arView.scene.anchors.append(boxAnchor)

Step 4: Programmatically Create AR Content

Instead of using Reality Composer, you can create 3D content directly in code:

import SwiftUI
import RealityKit
import ARKit

struct ARViewContainer: UIViewRepresentable {
    func makeUIView(context: Context) -> ARView {
        let arView = ARView(frame: .zero)

        // Create a simple box
        let mesh = MeshResource.generateBox(size: 0.1)
        let material = SimpleMaterial(color: .blue, isMetallic: false)
        let boxEntity = ModelEntity(mesh: mesh, materials: [material])

        // Create an anchor entity
        let anchorEntity = AnchorEntity(plane: .horizontal, classification: .any)
        anchorEntity.addChild(boxEntity)

        // Add the anchor to the scene
        arView.scene.anchors.append(anchorEntity)

        return arView
    }

    func updateUIView(_ uiView: ARView, context: Context) {}
}

Explanation:

• A blue box is programmatically created using MeshResource and SimpleMaterial.
• An AnchorEntity is used to position the box on a horizontal plane detected by ARKit.

Adding User Interaction

Step 1: Implement Tap Gesture

Add a tap gesture recognizer to enable user interactions with the AR content:

import SwiftUI
import RealityKit
import ARKit
import UIKit

struct ARViewContainer: UIViewRepresentable {
    func makeUIView(context: Context) -> ARView {
        let arView = ARView(frame: .zero)

        // Create a simple box
        let mesh = MeshResource.generateBox(size: 0.1)
        let material = SimpleMaterial(color: .blue, isMetallic: false)
        let boxEntity = ModelEntity(mesh: mesh, materials: [material])
        boxEntity.name = "myBox" // Give the entity a name

        // Create an anchor entity
        let anchorEntity = AnchorEntity(plane: .horizontal, classification: .any)
        anchorEntity.addChild(boxEntity)

        // Add the anchor to the scene
        arView.scene.anchors.append(anchorEntity)

        // Add tap gesture recognizer
        let tapGesture = UITapGestureRecognizer(target: context.coordinator, action: #selector(Coordinator.handleTap(recognizer:)))
        arView.addGestureRecognizer(tapGesture)
        
        context.coordinator.arView = arView // Store a reference to arView

        return arView
    }

    func updateUIView(_ uiView: ARView, context: Context) {}

    func makeCoordinator() -> Coordinator {
        Coordinator(self)
    }

    class Coordinator: NSObject {
        var parent: ARViewContainer
        weak var arView: ARView? // Add a weak reference to arView

        init(_ parent: ARViewContainer) {
            self.parent = parent
        }

        @objc func handleTap(recognizer: UITapGestureRecognizer) {
            guard let arView = arView else { return } // Safely unwrap arView
            let tapLocation = recognizer.location(in: arView)

            let results = arView.raycast(from: tapLocation, allowing: .estimatedPlane, alignment: .horizontal)

            if let firstResult = results.first {
                let worldPosition = SIMD3(firstResult.worldTransform.columns.3.x, firstResult.worldTransform.columns.3.y, firstResult.worldTransform.columns.3.z)
                
                // Create a new box at the tap location
                let mesh = MeshResource.generateBox(size: 0.05)
                let material = SimpleMaterial(color: .red, isMetallic: false)
                let newBox = ModelEntity(mesh: mesh, materials: [material])
                newBox.position = worldPosition

                // Add the new box to the scene
                let anchorEntity = AnchorEntity()
                anchorEntity.addChild(newBox)
                arView.scene.anchors.append(anchorEntity)
            } else {
                // Alternative approach using entity(at:) - Less accurate
                let tappedEntities = arView.entities(at: tapLocation)
                for entity in tappedEntities {
                    if entity.name == "myBox" {
                        print("Tapped on myBox")
                        entity.removeFromParent() // Example action: remove tapped box
                        return
                    }
                }
            }
        }
    }
}

Key Improvements and Explanations:

• Safely Unwrapping arView: Uses guard let arView = arView else { return } to ensure arView is not nil before using it, preventing potential crashes.
• Weak Reference to arView: Added weak var arView: ARView? to avoid retain cycles.
• Storing arView Reference: The arView instance is now correctly stored and accessible in the Coordinator.
• Two Tap Handling Methods:
  • Ray Casting for Plane Detection: The primary method is to use arView.raycast to detect planes. If a plane is detected, a new box is created at that location.
  • Entity Detection as a Fallback: The secondary method is to use arView.entities(at:) to directly detect if the tapped location contains the named entity (“myBox”). This approach serves as a fallback, particularly useful if ray casting doesn’t detect a plane. This method is less accurate than ray casting as it depends on the entity’s visual representation directly being tapped.
• Removing Tapped Box (Action Example): When “myBox” is tapped using the fallback method, it is removed from the scene using entity.removeFromParent().

Step 2: Add SwiftUI Controls

Overlay SwiftUI controls on the ARView using ZStack to create more complex AR interactions.

import SwiftUI
import RealityKit

struct ContentView : View {
    @State private var isButtonPressed = false

    var body: some View {
        ZStack {
            ARViewContainer().edgesIgnoringSafeArea(.all)

            VStack {
                Spacer()
                Button(action: {
                    isButtonPressed.toggle()
                }) {
                    Text(isButtonPressed ? "Button Pressed" : "Press Me")
                        .padding()
                        .background(Color.blue)
                        .foregroundColor(.white)
                        .cornerRadius(10)
                }
                .padding()
            }
        }
    }
}

#if DEBUG
struct ContentView_Previews : PreviewProvider {
    static var previews: some View {
        ContentView()
    }
}
#endif

Explanation:

• A ZStack is used to overlay the SwiftUI button on top of the ARViewContainer.
• The button’s state is managed using @State, updating the UI when pressed.

Advanced AR Features with RealityKit

Physics Simulation

Add realistic physics to your AR objects using RealityKit’s physics engine:

import RealityKit
import ARKit

func addPhysics(to entity: Entity) {
    let physicsBody = PhysicsBodyComponent(
        shapes: [ShapeResource.generateBox(size: [0.1, 0.1, 0.1])],
        mass: 0.1,
        mode: .dynamic
    )
    entity.components.set(physicsBody)

    let collisionComponent = CollisionComponent(
        shapes: [ShapeResource.generateBox(size: [0.1, 0.1, 0.1])],
        mode: .default,
        filter: .sensor
    )
    entity.components.set(collisionComponent)
}

Spatial Audio

Incorporate spatial audio to make your AR environment more immersive:

import RealityKit
import AVFoundation

func addSpatialAudio(to entity: Entity) {
    guard let audioFile = Bundle.main.url(forResource: "ambient_sound", withExtension: "mp3") else { return }

    do {
        let audioSource = try AVAudioPlayerNode()
        let audioFileObject = try AVAudioFile(forReading: audioFile)
        
        let audioFormat = audioFileObject.processingFormat
        audioSource.scheduleFile(audioFileObject, at: nil)
        audioSource.play()

        let spatialAudioComponent = SpatialAudioComponent(audioSource: audioSource)
        entity.components.set(spatialAudioComponent)

    } catch {
        print("Error loading audio: (error)")
    }
}

Conclusion

SwiftUI and RealityKit together offer a streamlined approach to building augmented reality experiences on Apple devices. SwiftUI simplifies UI development while RealityKit provides robust AR rendering and interaction capabilities. By following the steps and examples outlined in this blog post, developers can create engaging and immersive AR applications, pushing the boundaries of what’s possible in augmented reality.