Drone VR

Overview

Drone VR is a virtual reality training platform that teaches students how to unpack, assemble, and prepare the Innocon MiniFalcon drone for flight. The physical MiniFalcon costs approximately $600,000, making hands-on training with the actual hardware impractical for most UAS programs. This VR trainer provides a risk-free environment where students can practice the full assembly workflow on Meta Quest headsets before ever touching the real drone.

Funded by the National Science Foundation through the National Center for Autonomous Technologies (NCAT) and developed at the Shenandoah Center for Immersive Learning, the project uses Unity with XR Interaction Toolkit to deliver socket-based assembly mechanics where each drone part must be correctly identified and placed in the right location.

Role Summary

• Sole developer for v2.0, taking over as lead after v1.0 was co-developed. Designed and implemented the entire configurable assembly system architecture from scratch, including the AssemblyStepProcessHandler state machine, AssemblyStepHandler orchestration layer, and all custom XR components. Built the Key/Lock interaction constraint framework, multi-mode training system with ScriptableObject-based settings, dual-stopwatch timing engine, and JSON data export pipeline. Migrated the project from Unity 2022 to Unity 6, resolving package compatibility issues including reverting XRITK to 2.5.2 after upgrade-breaking changes. Created the visual affordance handler, tooltip system, and report generation infrastructure.

Non-Technical Summary

The second version of Drone VR is a major upgrade that transforms the platform from a basic assembly exercise into a full training and assessment tool. Students can now train in three different ways: a Learning mode that provides step-by-step guidance with visual hints showing exactly where each part goes, a Practice mode where students can request help when they get stuck, and an Assessment mode that tests their knowledge under a time limit with no assistance.

The system now tracks everything a student does: how long each assembly step takes, whether they needed hints, and their total completion time. This data is automatically saved as a report file that instructors can review, making it possible to evaluate student progress and identify areas where they need more practice.

Behind the scenes, the entire system was rebuilt to be more flexible. Each step of the assembly process is defined as a separate configuration file, making it easy to add new steps, change instructions, or modify the assembly sequence without touching the code. The interaction system was also improved with a "key and lock" mechanism that ensures parts can only be placed in the correct order, for example, you can't install the wing before opening the crate.

Highlights

• Architected and implemented a configurable, data-driven assembly step system with ScriptableObject-based step definitions, multi-condition completion tracking, and three training modes (Learning, Practice, Assessment) for VR drone assembly training
• Engineered a custom Key/Lock interaction constraint framework extending XR Interaction Toolkit, restricting grab and socket interactions to authorized object combinations via an IKeychain/Lock/Key architecture
• Designed and built XRConfigurableInteractable and XRConfigurableSocketInteractor components supporting one-handed and two-handed interactions, PreInstallation/Installation/Removal socket states, and automatic completion evaluation
• Migrated entire project from Unity 2022 LTS to Unity 6 (6000.1.0f1), upgrading Meta XR SDK to v77, OpenXR to 1.15.1, and URP to 17.2.0 while resolving XRITK compatibility issues
• Built automated JSON data export pipeline logging per-step timing, completion events, user actions, and scene settings via Newtonsoft.Json for student performance analytics

Quick Highlights

• Complete architecture rewrite with data-driven, configurable assembly step system
• Three training modes: Learning, Practice, and Assessment with per-mode settings
• Custom Key/Lock interaction constraint system preventing incorrect assembly sequences
• Extended XR Interaction Toolkit with XRConfigurableInteractable and XRConfigurableSocketInteractor
• Automated JSON export of student performance metrics (timing, completion, actions)
• Migrated to Unity 6 with Meta XR SDK v77 and OpenXR 1.15.1

Technical Breakdown

Configurable Assembly Step System

The core of v2.0 is the AssemblyStepProcessHandler, a state machine that manages progression through an ordered list of AssemblyStepConfiguration objects. Each configuration references an AssemblyStepScriptableObject (containing stepTitle, stepInstructions, stepHint) and a list of AssemblyStepHandler components. The process handler tracks previous, current, and next step indices and orchestrates configuration, completion checking, and advancement. Steps are configured with a delay via ConfigureAssemblyStep() and advance via GoToNextAssemblyStep(). A hint system with OnHintButton() enables silhouettes, tooltips, and affordances progressively.

Custom XR Interaction Components

XRConfigurableInteractable extends XRGrabInteractable and implements IKeychain. It supports multi-condition completion evaluation through three condition layers: Primary (SelectEntered, SelectEnteredAndExited, SelectExited), Secondary (EnteredHover, ExitedHover), and Trinary (EnteredXRSocket, ExitedXRSocket). The component tracks selecting interactors for two-handed validation, manages rigidbody freeze/unfreeze during interaction, and resets objects to starting position on floor collision. Each interactable holds a list of Key ScriptableObjects as its keychain.

XRConfigurableSocketInteractor extends XRSocketInteractor with three interaction states via ConfigurableSocketInteractorType: PreInstallation, Installation, and Removal. It tracks socket interaction completion and controls ghost mesh visualization based on scene settings. Sockets are wrapped in XRConfigurableSocketCollection which tracks availability and completion state alongside the socket's MeshRenderer reference.

Key/Lock Constraint System

The IKeychain interface defines Contains(Key key). Keychain implements it with a HashSet for O(1) lookup. Lock holds required keys and validates via CanUnlock(IKeychain), requiring ALL keys present. XRLockSocketInteractor overrides CanHover() and CanSelect() to validate the interactable's keychain against the socket's lock before allowing interaction. This enforces assembly order: parts gain keys as earlier steps complete.

Multi-Mode Training via ScriptableObjects

SceneSettingsScriptableObject defines per-mode configuration: allowAssemblyStepTiming, allowObjectAssemblyTiming, trackAssemblyStepCompletion, showAssemblyInstructions, plus enums for HintOption (NoHints, AlwaysShowHints, AllowPromptedHints), OutlineOption, HapticsOption, and FlawType (RandomFlaw, StructuralFlaw, ElectricalFlaw, NoFlaws). SettingsHandler routes these to the appropriate handlers and ReportHandler.

Assessment and Timing

TimingHandler manages two Stopwatch instances, one for full assembly duration and one for the current step. ReturnStopwatchSpan() captures elapsed time and optionally restarts. AssemblyAssessmentHandler validates completion within a configurable time limit (15-120 minutes) and triggers data export via StartAssemblyAssessment().

Data Export Pipeline

DataExportHandler is the base class handling file path preparation via Application.persistentDataPath, folder creation, and filename generation with numeric suffixes for duplicates. ReportHandler extends it, accumulating an AssemblyLog object with lists for sceneSettings, userActions, assemblyStepTiming, and assemblyStepCompletion. On OnApplicationQuit() or explicit export call, it serializes to JSON via Newtonsoft.Json.

UI and Visual Feedback

UIHandler manages multiple Canvas elements, updating instruction displays from AssemblyStepScriptableObject data and toggling mode-specific UI. AffordanceHandler provides hover feedback via coroutine-driven material lerping between original and hover materials. TooltipHandler positions world-space Canvas tooltips above interactables with billboard rotation facing the player camera.

Systems Used

• Configurable Assembly Step System - Data-driven assembly progression engine using ScriptableObject step definitions with configurable completion conditions
• Key/Lock Interaction Constraint System - Custom IKeychain/Lock/Key architecture restricting XR socket and grab interactions to authorized object combinations
• XR Configurable Interactable Framework - Extended XRGrabInteractable with multi-condition completion tracking, two-handed interaction support, and keychain integration
• XR Configurable Socket Interactor - Custom socket interactor with PreInstallation, Installation, and Removal states and ghost visualization control
• Multi-Mode Training System - Three simulation modes (Learning, Practice, Assessment) with per-mode ScriptableObject settings controlling hints, outlines, haptics, and timing
• Assembly Timing and Assessment Engine - Dual-stopwatch timing system with configurable time limits for step-level and full-assembly performance tracking
• JSON Data Export Pipeline - Automated session logging and JSON export of assembly metrics, step timing, user actions, and scene settings via Newtonsoft.
• Json
• Visual Affordance Handler - Material lerping system providing hover feedback through coroutine-driven material transitions on interactable objects
• World-Space Tooltip System - Billboard-oriented tooltips positioned above interactables with camera-facing rotation for contextual guidance

Deep Dive

Architecture Rewrite Motivation

Version 1.0 proved the concept but had rigid, hard-coded interactions that were difficult to extend. Adding a new assembly step required modifying scene objects and scripts directly. The v2.0 rewrite introduced a fully data-driven architecture where assembly steps, interaction constraints, and training modes are all configured through ScriptableObjects and serialized collections, making the system extensible without code changes.

Assembly Step State Machine

The AssemblyStepProcessHandler is the central state machine. It holds an ordered array of AssemblyStepConfiguration objects, each containing a reference to an AssemblyStepScriptableObject (title, instructions, hint text), a list of AssemblyStepHandler components, and a UnityEvent for triggering additional step-specific functions. The process handler tracks three indices (previous, current, next) and manages the lifecycle: enable step components → configure after delay → poll for completion → advance.

Each AssemblyStepHandler manages the XR objects for its step. It holds parallel lists of XRConfigurableInteractableCollection objects (grouped by type) and XRConfigurableSocketCollection objects (separated into PreInstallation, Installation, and Removal lists). When configured via ConfigureXRObjects(), it enables all interactables and sockets, starts timing if enabled, and begins polling CheckAssemblyStepStatus() each frame. Completion requires ALL interactables and ALL sockets in the step to report their conditions met.

Multi-Condition Interaction Evaluation

The condition evaluation system in AssemblyStepHelper supports three layers of completion conditions evaluated independently: Primary conditions track XRGrabInteractable select states (SelectEntered, SelectEnteredAndExited, SelectExited). Secondary conditions track hover states (EnteredHover, ExitedHover). Trinary conditions track socket states (EnteredXRSocket, ExitedXRSocket). Each XRConfigurableInteractable evaluates all three layers, and a step only completes when every interactable and socket in the step reports all configured conditions met.

This layered approach enables complex interaction sequences. For example, a "pick up and place" step might require SelectEntered (user grabbed the part) AND EnteredXRSocket (part was placed in socket). A "remove and inspect" step might require SelectEntered AND ExitedXRSocket. The two-handed interaction mode validates that two distinct interactors (left and right hand) are simultaneously selecting the object.

Key/Lock Constraint Architecture

The Key/Lock system enforces assembly ordering without hard-coding dependencies. Key is a ScriptableObject representing a capability. IKeychain is an interface with a single method: Contains(Key key). XRConfigurableInteractable implements IKeychain and holds a list of keys. Lock holds required keys and validates via CanUnlock(IKeychain), requiring ALL keys present.

XRLockSocketInteractor overrides CanHover() and CanSelect() to check whether the approaching interactable's keychain satisfies the socket's lock. This means a wing spar socket can require a "crate-opened" key, which only appears on interactables after the crate-opening step completes. The system is composable: multiple keys can gate a single socket, and a single interactable can hold multiple keys.

ScriptableObject-Driven Configuration

SceneSettingsScriptableObject assets define per-mode behavior through boolean flags and enums. Three assets exist: Learning (instructions always shown, hints always visible, outlines always on, haptics enabled, no timing), Practice (instructions shown, hints on prompt, outlines on prompt, haptics on prompt, timing enabled), and Assessment (no instructions, no hints, no outlines, timing with limits, flaws enabled). SettingsHandler loads the appropriate asset and exposes getters that other handlers query at runtime.

AssemblyStepScriptableObject assets hold per-step content: stepTitle, stepInstructions, and stepHint. These feed into UIHandler.SetAssemblyDisplay() which updates the instruction panel Canvas. New steps are added by creating a new ScriptableObject asset and adding an AssemblyStepConfiguration entry, requiring no code modifications.

Performance Tracking and Export

TimingHandler uses two System.Diagnostics.Stopwatch instances for precise timing. The step stopwatch starts when a step is configured and stops on completion. The assembly stopwatch runs for the entire session. ReturnStopwatchSpan() captures the current elapsed time and optionally restarts the stopwatch in one atomic operation.

ReportHandler extends DataExportHandler and accumulates an AssemblyLog throughout the session. It records: scene settings with timestamps, per-step completion events, per-step timing, user-prompted hint events, and total assembly duration. On OnApplicationQuit(), it serializes the log to JSON using Newtonsoft.Json and writes to Application.persistentDataPath. Filenames include the random 4-digit username from ReportNameHandler with numeric suffixes for duplicates.

Unity 6 Migration

The migration from Unity 2022 to Unity 6 (6000.1.0f1) required upgrading all packages: Meta XR SDK to v77.0.0, OpenXR to 1.15.1, URP to 17.2.0, and Input System. The XR Interaction Toolkit was initially upgraded but had to be reverted to 2.5.2 due to breaking changes (commit 7bc809f: "Revert to XRITK 2.5.2 so everything doesn't break"). Render pipeline changes and project settings were updated across multiple commits. The migration preserved all existing functionality while enabling Unity 6 features.

Visual Feedback Systems

AffordanceHandler provides hover feedback by lerping between an original material and a configurable hover material using coroutines. The lerp duration is configurable for smooth transitions. TooltipHandler manages world-space Canvas elements positioned above interactables with a configurable Y-offset. Tooltips use billboard rotation to always face the player camera, ensuring readability from any angle. The Drone_Silhouette_Combined_Model provides ghost outlines at assembly positions, controlled by socket ghost mesh renderers that toggle based on scene settings and hint state.

Handler Architecture

The codebase follows a handler/manager pattern where each system is encapsulated in a dedicated MonoBehaviour: SimulationHandler (master orchestrator with references to all other handlers), SettingsHandler (mode-based configuration), UIHandler (Canvas management), TimingHandler (stopwatches), ReportHandler (data logging), AssemblyStepProcessHandler (step progression), and AssemblyStepHandler (per-step XR object management). SimulationHandler holds references to left/right XRDirectInteractor components and the player camera, distributing them to subsystems as needed.