visionOS

visionOS support and spatial computing features in the LiveKit Swift SDK

The LiveKit Swift SDK provides support for Apple Vision Pro and visionOS, enabling immersive video conferencing experiences in spatial computing environments.

Platform Support

Minimum version: visionOS 1.0+

visionOS support enables building spatial computing experiences with LiveKit on Apple Vision Pro.

Capabilities

visionOS shares most capabilities with iOS while running in a spatial computing environment.

Supported Features

• ✅ Connect to LiveKit rooms
• ✅ Receive and render video tracks
• ✅ Receive and play audio tracks
• ✅ Persona camera (systemPreferredCamera)
• ✅ Microphone capture
• ✅ Data channels (send/receive)
• ✅ Room events and participant management
• ✅ VideoView rendering
• ✅ SwiftUI components
• ✅ Spatial audio positioning

Limited Features

• ⚠️ Screen sharing: Limited compared to iOS/macOS
• ⚠️ Camera switching: Only Persona camera available
• ⚠️ Metal rendering: Some Metal features may differ

VideoView

The SDK provides a UIView-based VideoView for rendering video tracks in visionOS:

import LiveKit
import UIKit

class RoomViewController: UIViewController {
    lazy var videoView: VideoView = {
        let videoView = VideoView()
        view.addSubview(videoView)
        videoView.layoutMode = .fit
        return videoView
    }()

    override func viewDidLoad() {
        super.viewDidLoad()
        
        Task {
            try await connectToRoom()
        }
    }
}

Video Rendering

visionOS supports video rendering:

• AVSampleBuffer Rendering (recommended): Works reliably on visionOS
• Metal Rendering: May have platform-specific considerations

videoView.renderMode = .sampleBuffer // Recommended for visionOS

Layout Options

// Control how video fits within the view
videoView.layoutMode = .fit // .fit recommended for spatial layouts

// Mirror video
videoView.mirrorMode = .auto

Camera Support

visionOS uses the Persona camera system for capturing the user’s presence.

Persona Camera

The Persona camera represents the user in a spatial computing context:

import AVFoundation

// The Persona camera is available as systemPreferredCamera
if let personaCamera = AVCaptureDevice.systemPreferredCamera {
    print("Persona camera available: \(personaCamera.localizedName)")
    
    let options = CameraCaptureOptions(
        dimensions: .h720_169,
        fps: 30
    )
    
    try await room.localParticipant.setCamera(
        enabled: true,
        captureOptions: options
    )
}

Device Discovery

visionOS has a simplified device discovery:

// Get available cameras
let devices = try await CameraCapturer.captureDevices()

// On visionOS, this typically returns [systemPreferredCamera]
for device in devices {
    print("Camera: \(device.localizedName)")
}

See DeviceManager.swift:117 for visionOS implementation:

#elseif os(visionOS)
// For visionOS, there is no DiscoverySession so return the Persona camera if available.
let devices: [AVCaptureDevice] = [.systemPreferredCamera].compactMap { $0 }

Audio Support

Spatial Audio

visionOS supports spatial audio, providing an immersive audio experience:

import AVFoundation

// Audio session configuration for visionOS
let session = AVAudioSession.sharedInstance()
try session.setCategory(.playAndRecord, mode: .videoChat, options: [.mixWithOthers])
try session.setActive(true)

Audio Session Management

visionOS uses AVAudioSession similar to iOS:

// Automatic configuration (default)
AudioManager.shared.audioSession.isAutomaticConfigurationEnabled = true

// Manual configuration
AudioManager.shared.audioSession.isAutomaticConfigurationEnabled = false
let session = AVAudioSession.sharedInstance()
try session.setCategory(.playAndRecord, mode: .videoChat)
try session.setActive(true)

See AudioSessionConfiguration.swift:17 for available configurations.

Microphone Capture

// Enable microphone
try await room.localParticipant.setMicrophone(enabled: true)

// Custom audio settings
let audioOptions = AudioCaptureOptions(
    echoCancellation: true,
    noiseSuppression: true
)

SwiftUI Integration

visionOS works great with SwiftUI for spatial computing experiences:

import SwiftUI
import LiveKit

struct SpatialVideoView: View {
    @StateObject var room = Room()
    @State private var participants: [RemoteParticipant] = []

    var body: some View {
        ZStack {
            // Background
            Color.black.ignoresSafeArea()
            
            // Participant videos in spatial layout
            HStack(spacing: 20) {
                ForEach(participants, id: \.sid) { participant in
                    if let videoTrack = participant.videoTracks.first?.track {
                        SwiftUIVideoView(videoTrack)
                            .frame(width: 400, height: 300)
                            .cornerRadius(12)
                    }
                }
            }
        }
        .onAppear {
            Task {
                try await room.connect(url: url, token: token)
                participants = Array(room.remoteParticipants.values)
            }
        }
    }
}

Volumetric Windows

visionOS supports volumetric windows for 3D content:

import SwiftUI

@main
struct MyVisionApp: App {
    var body: some Scene {
        WindowGroup {
            ContentView()
        }
        
        // Volumetric window for immersive video
        WindowGroup(id: "videoCall") {
            SpatialVideoView()
        }
        .windowStyle(.volumetric)
        .defaultSize(width: 800, height: 600, depth: 400, in: .points)
    }
}

Immersive Spaces

Create fully immersive experiences:

import SwiftUI
import RealityKit

struct ImmersiveVideoSpace: View {
    @StateObject var room = Room()
    
    var body: some View {
        RealityView { content in
            // Add 3D content with video textures
        }
        .onAppear {
            Task {
                try await room.connect(url: url, token: token)
            }
        }
    }
}

// Register immersive space
@main
struct MyVisionApp: App {
    var body: some Scene {
        ImmersiveSpace(id: "videoSpace") {
            ImmersiveVideoSpace()
        }
    }
}

Data Channels

Data channels work fully on visionOS:

// Send data
try await room.localParticipant.publish(data: data, options: DataPublishOptions(
    reliable: true
))

// Receive data
func room(_: Room, participant: RemoteParticipant, didReceiveData data: Data, forTopic topic: String) {
    print("Received data: \(data.count) bytes")
}

Permissions

visionOS requires permissions for camera and microphone.

Info.plist Entries

Add these keys to your Info.plist:

<key>NSCameraUsageDescription</key>
<string>This app needs camera access to show your Persona in video calls</string>

<key>NSMicrophoneUsageDescription</key>
<string>This app needs microphone access for audio calls</string>

Requesting Permissions

import AVFoundation

// Request camera permission (Persona)
let cameraStatus = AVCaptureDevice.authorizationStatus(for: .video)
if cameraStatus == .notDetermined {
    let granted = await AVCaptureDevice.requestAccess(for: .video)
    print("Camera permission: \(granted)")
}

// Request microphone permission
let micStatus = AVCaptureDevice.authorizationStatus(for: .audio)
if micStatus == .notDetermined {
    let granted = await AVCaptureDevice.requestAccess(for: .audio)
    print("Microphone permission: \(granted)")
}

Screen Sharing

Screen sharing on visionOS has different considerations than iOS:

// Screen sharing may work differently on visionOS
// Check availability first
do {
    try await room.localParticipant.setScreenShare(enabled: true)
} catch {
    print("Screen sharing not available: \(error)")
}

Screen sharing capabilities on visionOS may be limited compared to iOS due to the spatial computing environment.

Network Considerations

visionOS supports standard networking:

// WebSocket connections work normally
try await room.connect(url: url, token: token)

// Network quality monitoring
func room(_: Room, participant: RemoteParticipant, didUpdateConnectionQuality quality: ConnectionQuality) {
    print("Connection quality: \(quality)")
}

See WebSocket.swift:30 for visionOS network handling.

Building and Testing

# Build for visionOS Simulator
xcodebuild build -scheme LiveKit -destination 'platform=visionOS Simulator'

# List available simulators
xcrun simctl list devices visionOS

Use Cases

visionOS enables unique spatial computing experiences:

• Spatial meetings: Video calls in 3D space
• Collaborative work: Shared immersive environments
• Virtual events: Conferences in spatial environments
• Education: Immersive learning experiences
• Healthcare: Spatial telemedicine consultations

Performance Considerations

1. Video decoding: Hardware-accelerated on Apple Vision Pro
2. Multiple streams: Can handle multiple video streams simultaneously
3. Spatial audio: Optimized for immersive audio experiences
4. Battery life: Consider power management for extended sessions

Thread Safety

VideoView must be accessed from the main thread on visionOS.

// Correct: Update VideoView on main thread
DispatchQueue.main.async {
    videoView.track = track
}

// Or use MainActor
Task { @MainActor in
    videoView.track = track
}

Limitations

Platform-Specific

1. Single camera: Only Persona camera available (no front/back switching)
2. Metal features: Some Metal APIs may differ from iOS
3. Screen sharing: Different implementation than iOS/macOS

API Considerations

• No AVCaptureDevice.DiscoverySession (uses systemPreferredCamera instead)
• Spatial audio requires different configuration
• Immersive spaces require RealityKit integration

Best Practices

1. Use SwiftUI: Leverage SwiftUI for spatial layouts
2. Optimize for immersion: Design for 3D space
3. Handle permissions early: Request camera/mic permissions on launch
4. Test on device: Simulator may not reflect actual behavior
5. Consider spatial audio: Position audio sources appropriately

Example Projects

• Swift SDK Examples Collection

Next Steps

iOS Platform

Compare with iOS capabilities

SwiftUI Components

Use SwiftUI components in your app