Extended Reality

Gesture Interface

Kinect-powered gesture recognition display for interactive content browsing

A gesture-driven interactive display system built in Unity that integrates Microsoft Kinect v2 skeleton tracking with UI Toolkit to deliver an immersive, multi-modal content browsing experience. Users navigate paired content displays through physical gestures, with automatic idle slideshows and crowd modes for passive viewing.

Explore details

My role: Lead XR Developer
Period: Jul 2022 – Dec 2025
Platforms: Windows
Status: delivered

Tools & technologies

C#UnityKinect SDKUniversal Render PipelineUI ToolkitCinemachineTimelineInput System

Impact & Results

Eliminated tight coupling across the entire codebase by introducing dependency injection and event-driven communication, making every component independently testable and replaceable. The startup pipeline's graceful fallback design ensures the application runs in degraded mode (local content, no Kinect) rather than crashing when infrastructure is unavailable. The SafeEvent pattern eliminated a class of runtime crashes where one faulty event subscriber could bring down the entire application.

Overview

Gesture Interface is an interactive display system that uses Microsoft Kinect v2 skeleton tracking to let visitors browse content through physical gestures. When a user steps into the detection zone, paired content displays respond to their body movements, enabling hands-free navigation through images, text, and media. When no one is present, the system automatically transitions to an idle slideshow or crowd mode for passive viewing.

Developed at the Shenandoah Center for Immersive Learning for deployment in public-facing exhibition spaces, the project has evolved through three major versions: from a proof-of-concept to a production system with event-driven architecture, dependency injection, and extracted reusable packages. It is built in Unity with UI Toolkit and designed for unattended operation on Windows display hardware.

Role Summary

Led the complete architectural redesign as sole developer. Designed the event-driven state management system with AppFlowBehaviour. Created the dependency injection framework with ServiceRegistry supporting [AutoRegister] and [Inject] attributes. Built the StartupProcessProvider sequential pipeline. Refactored all mode handlers, content services, and body rig systems to use interface-based composition.

Non-Technical Summary

Under the hood, this version was about making the system professional-grade. Previously, the different parts of the application talked to each other directly: the gesture detector would call the display handler, which would call the content loader. If any piece changed, everything connected to it had to change too.

In v3.0, the entire system was redesigned so that components communicate through events and interfaces rather than direct connections. Think of it like upgrading from a telephone tree (where each person calls the next) to a bulletin board system (where anyone can post updates and anyone can read them). This makes it much easier to add new features, fix bugs, or swap out components without breaking everything else.

Highlights

Architected an event-driven state machine (AppFlowBehaviour) with queued mode transitions, transition scoping, and SafeEvent invocation to eliminate race conditions and exception cascades across 3 runtime modes
Designed and implemented a dependency injection framework using [AutoRegister] and [Inject] attributes for scene-scoped service resolution
Built a sequential startup pipeline with 7 ordered initialization steps (service validation, environment checks, config loading, server health, content processing, and Kinect initialization) with graceful fallbacks at each stage
Refactored all mode handlers to interface-driven composition with ModeOrchestrator coordinating IAppCommands and IAppActivity for fully decoupled mode lifecycle management

Quick Highlights

Event-driven state machine with SafeEvent preventing exception cascades
Dependency injection via [AutoRegister] and [Inject] attributes
Sequential 7-step startup pipeline with graceful fallbacks
Interface-driven mode orchestration with IAppCommands/IAppActivity
Thread-safe async/await with UnityMainThread synchronization

Technical Breakdown

Event-Driven State Machine: AppFlowBehaviour implements IAppFlow and manages runtime mode transitions through an event-driven architecture. Mode change requests are queued during active transitions, preventing race conditions. TransitionScope tracks ownership of active transitions, and when a handler begins a transition, it acquires a scope that must be disposed before new transitions can proceed. SafeEvent wraps event invocation to catch and log exceptions from individual subscribers without breaking the event chain, preventing a single faulty handler from crashing the entire application. Events include ModeChanged, TransitionChanged, and InteractionOccurred.

Dependency Injection: The ServiceRegistry provides scene-scoped dependency injection. Services are registered using [AutoRegister] attributes on MonoBehaviours, which are discovered at scene load. Consumer scripts use [Inject] attributes on fields to receive resolved services. The registry validates that all required services are registered during the startup pipeline, failing fast with clear error messages if dependencies are missing. This replaced the previous pattern of direct GetComponent and singleton references throughout the codebase.

Sequential Startup Pipeline: StartupProcessProvider defines a 7-step ordered initialization sequence: (1) Validate Services: ensure all critical services are registered, (2) Validate Environment: check persistent storage paths, layer setup, and UI document availability, (3) Load Settings: async config initialization from AppConfigAsset ScriptableObject, (4) Check Server: HEAD request to content server for availability, (5) Process Content: load local content or trigger remote download based on server availability, (6) Kinect Init: initialize Kinect sensor with timeout fallback for environments without hardware, (7) Complete: transition to initial runtime mode. Each step reports progress to the InitializationUIHandler.

Config Registry: AppConfigProvider loads configuration from AppConfigAsset ScriptableObjects, supporting runtime reload via ConfigReloaded events. The AppConfig structure provides typed access to server settings (URL, credentials), content paths, body rig configuration, and idle timing parameters. Thread-safe async initialization ensures config is available before dependent systems start.

Mode Orchestrator: ModeOrchestrator implements both IAppCommands (for requesting mode changes and marquee transitions) and IAppActivity (for reporting user interaction). This centralizes mode coordination: InputHandler calls IAppCommands.RequestMarquee(), GestureHandler subscribes to IGestureHandler events, and the orchestrator manages the lifecycle. This replaced the previous pattern where handlers directly referenced each other.

Threading: UnityMainThread provides thread-safe async/await support, ensuring callbacks from async operations return to Unity's main thread. CancellationToken integration enables clean shutdown of long-running operations during mode transitions or application exit.

Systems Used

Event-Driven State Machine: AppFlowBehaviour with queued mode transitions, transition scoping, and SafeEvent invocation for decoupled runtime state management
Service Registry Dependency Injection: Scene-scoped service registration with [AutoRegister] and [Inject] attributes for loose coupling and testability
Sequential Startup Pipeline: Ordered startup process with validation, config loading, server checking, content processing, and Kinect initialization steps
Config Registry System: ScriptableObject-based configuration with runtime reload support, async initialization, and ConfigReloaded events
Content Service Architecture: Modular content pipeline with ContentParser, ContentMaterializer, ContentCarousel, and PersistentContentStorage components
Body Rig Priority System: Multi-body skeleton tracking with priority manager, zone interactors, interaction gates, and boundary policies
Marquee Animation Engine: Direction-aware slide transitions with PanelPair double-buffering, MarqueeAnimator coroutines, and transition scope locking
Mode Orchestrator Pattern: ModeOrchestrator implementing IAppCommands and IAppActivity interfaces for centralized mode handler coordination

Deep Dive

The v3.0 architectural overhaul was driven by the codebase having grown complex enough that tight coupling was causing cascading breakages during modifications.

Event System Design: The decision to use an event-driven architecture over a traditional state machine pattern was motivated by the need for multiple independent systems to react to state changes without knowing about each other. AppFlowBehaviour publishes ModeChanged events, and any number of subscribers, including UI handlers, body rig system, and content service, can respond independently. The SafeEvent wrapper was introduced after encountering a production issue where an exception in one subscriber's handler prevented all subsequent subscribers from receiving the event, effectively breaking the entire application from a single error.

Dependency Injection Approach: Rather than adopting a heavyweight DI framework like Zenject, a lightweight custom ServiceRegistry was built specifically for Unity's MonoBehaviour lifecycle. The [AutoRegister] attribute automatically discovers and registers services during scene initialization, while [Inject] resolves dependencies before Start() is called. This approach was simpler to debug than reflection-heavy frameworks and had minimal performance overhead. The registry supports scene-scoped lifetimes, which aligns with Unity's scene-based architecture.

Startup Pipeline Architecture: The sequential startup pipeline solved a recurring problem: initialization order dependencies. Previously, scripts relied on Unity's non-deterministic Awake()/Start() ordering, leading to intermittent null reference errors when services initialized in the wrong order. StartupProcessProvider makes the order explicit and adds validation at each step. The fallback design (skip Kinect if hardware is absent, use local content if server is unreachable) was critical for development environments where not all infrastructure is available.

Refactoring Strategy: The refactoring was executed in small, incremental commits rather than a single large rewrite. Each commit introduced one architectural change (e.g., "Implement EventSystem framework", "Add ServiceRegistry system", "Refactor BaseHandler to use [Inject]"), making it possible to bisect regressions. The body rig system, content service, and mode handlers were each refactored independently to use the new patterns, validating the approach at each step.

v3.0: Modular Event-Driven Architecture

v3.0 Overview

Gesture Interface v3.0 is a comprehensive architectural overhaul that introduced event-driven state management, dependency injection, and a sequential startup pipeline. All mode handlers were refactored to use interface-driven composition through a ModeOrchestrator, and the entire codebase was reorganized under a unified namespace. This version represents the maturation from a functional application into a well-architected, maintainable, and extensible system.

v2.0: Display System & Body Rig Architecture

v2.0 Highlights

Architected and built a complete body rig skeleton system with runtime rig generation, multi-body tracking, joint smoothing, and priority-based hand detection from Kinect v2 data
Designed a multi-mode content display supporting Gesture, Idle, and Crowd viewing modes with transition queuing, scope-based locking, and cross-dissolve animations
Migrated the entire UI layer from Canvas to Unity UI Toolkit, implementing marquee slide animations with double-buffered panel pairs for seamless content transitions
Built a content loading pipeline with JSON metadata parsing, async texture materialization, WebDAV-based remote downloads with retry logic, and fingerprint-based change detection
Engineered multi-display support enabling the application to drive multiple physical screens from a single Unity instance

v2.0 Quick Highlights

Full body skeleton visualization with runtime-generated rigs and scaled bone rendering
Three adaptive viewing modes: Gesture, Idle, and Crowd
UI Toolkit-based marquee animations with double-buffered panel pairs
WebDAV remote content downloads with retry logic and manifest caching
Priority-based multi-body hand tracking for gesture input arbitration
Multi-display output from a single Unity instance

v2.0 Overview

Gesture Interface v2.0 was a ground-up rebuild that transformed the prototype into a production-ready interactive display system. The project was re-initialized with Git LFS, migrated from Canvas UI to Unity UI Toolkit, and introduced a sophisticated body rig skeleton visualization system. Three distinct viewing modes (Gesture, Idle, and Crowd) allowed the system to adapt its presentation based on user engagement: Gesture for interactive paired content, Idle for automatic slideshow, and Crowd for fullscreen rotation. A complete content pipeline supported both local and remote content with async loading, and the Kinect integration was expanded with priority-based multi-body tracking and zone-based interaction detection.

v2.0 Role Summary

Led the complete rewrite of the project from the ground up. Designed and implemented the body rig skeleton system including SkeletonRenderer, BodyRigRuntimeRigBuilder, and BoneMap components. Built the multi-mode architecture with GestureHandler, IdleHandler, and CrowdHandler. Implemented the content loading pipeline with ContentParser, ContentMaterializer, and PersistentContentStorage. Developed the marquee animation system and UI Toolkit integration. Created the remote data download service with WebDAV support.

v2.0 Technical Breakdown

Body Rig System: The BodyRigSystemBehaviour orchestrates multi-body skeleton tracking. It receives tracked body data from KinectMediator, which bridges the Windows.Kinect SDK to the application through a KinectBodyToTrackedBodyAdapter. For each tracked body, a runtime skeleton rig is created using BodyRigRuntimeRigBuilder, which generates a hierarchy of joint GameObjects connected by SkeletonRenderer-drawn bones. A BoneMap maps Kinect joint types to visual bone segments. The BodyRigPriorityManager determines which body has "focus" based on highest hand position, and BodyRigZoneInteractor detects hand entry/exit from trigger zones. BodyRigSystemPolicy enforces interaction boundaries, while BodyRigInteractionGate guards input acceptance based on current application state.

Content Pipeline: Content follows a multi-stage pipeline. ContentParser reads JSON metadata files describing paired left/right content items with titles, captions, and image paths. ContentMaterializer handles async image loading and Unity texture creation. PersistentContentStorage manages local file enumeration, and ContentCarousel provides navigation control for cycling through content pairs. RemoteDataServiceBehaviour supports WebDAV-based file downloads with retry logic, exponential backoff, and manifest-based change detection.

Mode Handlers: Three mode handlers extend a common ModeHandlerBase. GestureHandler displays paired left/right content and drives marquee transitions via MarqueeAnimator. IdleHandler triggers automatic marquee transitions at configurable intervals when no user input is detected. CrowdHandler shows fullscreen content panels with fade-in/fade-out cross-dissolve animations, selecting 3 random non-repeating images per cycle.

UI System: The UI layer uses Unity UI Toolkit with three document instances (Left, Right, Middle) managed by UIToolkitHandler. GestureUIView implements a PanelPair pattern with "current" and "buffer" visual elements for smooth transitions. MarqueeAnimator slides content in from off-screen based on swipe direction. PulsingText provides configurable animated text for idle prompts.

Input System: InputHandler subscribes to Unity's new Input System for keyboard and gesture input, generating swipe events (left/right) and mode hotkeys. Input is throttled during transitions to prevent event stacking.

v2.0 Systems Used

Body Rig Skeleton System: Real-time multi-body skeleton tracking with runtime rig building, joint smoothing, and visual bone rendering from Kinect data
Content Loading Pipeline: JSON-based content parsing with async image materialization, persistent storage, and WebDAV remote download support
Marquee Animation System: Directional slide transition animations using panel pairs with current/buffer double-buffering for smooth content swaps
Multi-Mode State Machine: Runtime mode management supporting Gesture, Idle, and Crowd modes with transition queuing and scope-based locking
Body Rig Interaction System: Priority-based hand tracking with zone interactors and interaction gates for controlling gesture input acceptance
Remote Data Service: WebDAV-based file download system with retry logic, exponential backoff, and manifest-based change detection

v2.0 Impact Results

Transformed a basic swipe prototype into a production-quality interactive display system deployed for public use. The multi-mode architecture enabled the system to serve content effectively whether users are actively engaging, passively observing, or absent entirely. The body rig visualization provided intuitive user feedback, and the remote content pipeline eliminated the need for manual content updates on the display machine.

v2.0 Deep Dive

The v2.0 rewrite was motivated by the limitations of the prototype: tightly coupled scripts, Canvas UI that did not scale well across multiple displays, and a single interaction mode that left the screen blank when no users were present.

Project Re-initialization: The codebase was re-initialized with Git LFS to handle the growing number of binary assets (textures, prefabs, Unity scenes). New folder structures were established, assets were imported from a related HoloTour project, and the scene hierarchy was redesigned for the multi-mode architecture.

Body Rig Architecture: The most technically ambitious addition was the body rig system. Rather than simply tracking hand positions for gesture detection, v2.0 renders full skeleton visualizations of tracked bodies. BodyRigRuntimeRigBuilder dynamically creates hierarchies of joint GameObjects at runtime based on the Kinect skeleton topology. SkeletonRenderer draws scaled rectangle bones between joints (replacing the initial line renderer approach, which was changed to scaled rectangles for better visual presence). Joint smoothing reduces jitter from the Kinect sensor. The BodyRigPriorityManager tracks which of potentially multiple visible bodies has the "highest hand" and should receive input focus, a practical solution for public spaces where multiple people may be in the sensor's field of view simultaneously.

Multi-Mode Design: The three modes (Gesture, Idle, Crowd) each implement a common ModeHandlerBase and manage their own UI lifecycle. Mode transitions are queued and scope-locked: a transition in progress prevents new transitions from starting until the animation completes. This prevents race conditions where rapid user input or mode timer events could leave the UI in an inconsistent state. The IdleHandler monitors for user inactivity and automatically triggers marquee transitions at configurable intervals, while CrowdHandler uses a repetition-prevention algorithm to ensure the same image does not appear in consecutive display cycles.

Content Pipeline Design: Content is structured as JSON metadata files defining paired left/right items (title, caption, image path). This paired format was designed for the dual-display layout where left and right screens show complementary content. ContentMaterializer handles async texture loading, critical because Unity texture creation must happen on the main thread but file I/O should not block rendering. The ContentCarousel tracks the current position and provides bidirectional navigation.

Remote Download System: The RemoteDataServiceBehaviour implements a WebDAV-based content sync system. It downloads a manifest from the server, compares fingerprints against local files, and only downloads changed content. Retry logic with exponential backoff handles network interruptions, and the system supports partial success: if some files fail to download, the rest still become available.

UI Toolkit Migration: Moving from Canvas to UI Toolkit required rethinking the entire display layer. UI Toolkit's retained-mode model with USS styling replaced the Canvas GameObject hierarchy. The PanelPair abstraction (current + buffer elements) enables smooth transitions: the next content is pre-loaded into the off-screen buffer, the MarqueeAnimator slides both panels simultaneously, and then the roles swap. This double-buffering approach eliminates visual glitches during content changes.

v1.0: Original Kinect Prototype

v1.0 Highlights

Built an interactive gesture-recognition prototype integrating Microsoft Kinect v2 with Unity for a museum-style content display, enabling touchless navigation through physical hand swipes
Engineered real-time skeleton tracking with collider-based gesture detection, translating 3D hand positions into directional content transitions
Developed a multi-contributor integration effort, combining work from 4+ team members into a cohesive swipe-based content browsing experience
Implemented dynamic image loading from local storage with fade-in/fade-out transitions and animation-gated swipe availability to prevent input conflicts

v1.0 Quick Highlights

Touchless content browsing using Kinect v2 hand tracking
Collider-based gesture detection for left/right swipe recognition
Real-time skeleton joint visualization
Dynamic image loading from local file system
Animation-gated input to prevent double-swipes during transitions

v1.0 Overview

Gesture Interface v1.0 was the initial prototype for a touchless, gesture-driven content display system built for the Shenandoah Center for Immersive Learning (SCiL) lab. The application used a Microsoft Kinect v2 sensor to track user hand positions in 3D space, detecting left and right swipe motions through Unity trigger colliders. When a swipe was detected, the display transitioned between content images with fade animations. The prototype established the core interaction model (stand in front of a screen, wave your hand, and browse content) that would carry through all future versions.

v1.0 Role Summary

Developed the Kinect integration layer and swipe detection system. Collaborated with multiple team members (Tyler, Nate, Ann) to integrate separate workstreams (image loading, swipe mechanics, and body tracking) into a unified prototype. Owned the collider positioning and gesture detection tuning, iterating through multiple rounds of box position adjustments to achieve reliable swipe recognition.

v1.0 Technical Breakdown

Kinect Integration: The Kinect v2 SDK was imported directly into Unity with native Windows plugins. A KinectManager script polled the sensor for body frames, extracting joint positions for tracked skeletons. Hand joint positions were converted from Kinect's coordinate space to Unity world space for collision detection.

Gesture Detection: Left and right swipe gestures were detected using Unity trigger colliders positioned in 3D space. When a tracked hand entered a trigger zone, it registered a swipe event. The collider positions were iteratively tuned across multiple commits to achieve reliable detection without false positives. Animation state gating prevented swipe events from firing during active transitions.

Content Display: Images were loaded from a desktop folder at runtime. A ContentData structure held image references, and a DisplayHandler managed Canvas-based UI elements with CanvasGroup fade coroutines for smooth transitions between content items.

State Management: A simple RuntimeStatus enum tracked the application state. Swipe input was only accepted when the status was set to Gesture, preventing interactions during initialization or transitions.

v1.0 Systems Used

Kinect Body Tracking: Real-time skeleton joint tracking from Kinect v2 sensor for hand position detection
Collider-Based Gesture Detection: Unity trigger colliders positioned in 3D space to detect hand swipe gestures
Canvas UI Display System: Unity Canvas-based image display with CanvasGroup fade transitions for content presentation

v1.0 Impact Results

Successfully demonstrated touchless content navigation in a lab environment. The prototype validated the core interaction model, Kinect-based hand tracking driving content transitions, and established the foundation for subsequent versions. Integrated contributions from 4+ team members into a working demo within one academic year.

v1.0 Deep Dive

The v1.0 prototype was a collaborative effort that brought together several independent workstreams. The project began in July 2022 with a basic Unity setup and evolved through contributions from multiple team members working on different aspects of the system.

Integration Challenges: The primary technical challenge was merging separate implementations (Tyler's image loading, Ann's body tracking experiments, and Nate's UI contributions) into a cohesive application. An explicit "Begin combining Ann and Tyler's work" commit marks the integration point where these parallel efforts converged.

Collider Tuning: Achieving reliable swipe detection required extensive iteration on collider positions. The git history shows at least 4 sequential commits dedicated to adjusting box positions ("Updated Collider Positions", "Updated collider Positions Again", "Continuing to update box positions", "Box Positions change again"), reflecting the difficulty of mapping physical hand movements to reliable digital inputs without machine learning.

Input Gating: A critical usability improvement was animation-gated swipe availability. Without this, users could trigger multiple swipes during a single transition, causing content to skip. By linking input acceptance to animation state, the system ensured each gesture produced exactly one content transition.

Limitations: The prototype had several constraints that motivated the v2.0 rewrite: tightly coupled scripts with no dependency injection, Canvas-based UI that was difficult to scale, no remote content loading, and a single-mode interaction model with no idle or crowd viewing support.