Whirling Interface
Hand-based Motion Matching Selection
for Small Target on XR Displays
Juyoung Lee1, Seo Young Oh1, Minju Baeck1, Hui-Shyong Yeo2, Hyung-il Kim3,
Thad Starner4, Woontack Woo1,3
1KAIST UVR Lab.   2Huawei   3KI-ITC ARRC   4Georgia Tech
Contents
Whirling Interface
Whirling Concept
  • Goal:
    Develop a target selection interface suitable for daily use
  • Whirling Interface
    • Improved Interactivity
    • Consistent performance
  • Experiment
    • Ray casting vs Whirling
    • 48mm targets in {1m, 2m} distance
  • Discussion
    • Findings
    • Applications
    • Limitations
Introduction
Smartglasses is here
From Immersive headset with Spatial computing
to lightweight smartglasses to use in dailylife
MS HoloLens 2

MS HoloLens 2

Apple Vision Pro

Apple Vision Pro

Magic Leap 2

Magic Leap 2

z100

Vuzix Z100

Inmo Go2

Inmo Go2

Vuzix Shield

Vuzix Shield

Even Realities G1

Even Realities G1

Introduction
Goal
Create an interface for daily use AR
with small targets and bare-hand interaction
Why 'small targets'?
  • Minimize obstruction of the real world
  • Accommodate display's limited Field of View (FOV) in AR glasses
Why 'bare-hand' interaction?
  • Hand is the most common input modality
      Even the very early AR research suggested it[1]
  • No need to carry controllers all the time
  • Enables input even when hands are occupied by other objects
[1] Thad Starner, Steve Mann, Bradley Rhodes, Jeffrey Levine, Jennifer Healey, Dana Kirsch, Rosalind W. Picard, Alex Pentland; Augmented Reality through Wearable Computing. Presence: Teleoperators and Virtual Environments 1997; 6 (4): 386–398. doi: https://doi.org/10.1162/pres.1997.6.4.386
Related Work
Selection Methods for AR/VR

Ray Casting

Most common method

[Image]
Extended Fitts' law for 3D [3]
    Challenges:
  • Dependent on tracking[1]
    Tracking loss by out of FoV, Occlusion, Jitter, and others
  • Requires confirmation gesture
    Gesture affects performance[2]
  • Small targets are hard to select
    Following Fitts' Law[3]

Eye gaze pointing

Intuitive and ease of use

Photo: Apple, Vision Pro
Photo: Apple, Vision Pro
    Challenges:
  • Dependent on tracking
    Higher miss rate without pointer[4]
  • Requires confirmation gesture
  • Tracking issues on outdoors with sunlight

Multi-Modal/Step Input

Can complement methods

[Image]
Pinpointing [5]
    Challenges:
  • Requires more attention to use multi-modal
  • Have to remember multiple steps/modal combinations
  • Precise but slower
[1] Babu, S. V., Huang, H.-C., Teather, R. J. & Chuang, J.-H. Comparing the Fidelity of Contemporary Pointing with Controller Interactions on Performance of Personal Space Target Selection. 2022 IEEE Int. Symp. Mix. Augment. Real. (ISMAR) 00, 404–413 (2022).
[2] Dube, T. J., Ren, Y., Limerick, H., MacKenzie, I. S. & Arif, A. S. Push, Tap, Dwell, and Pinch: Evaluation of Four Mid-air Selection Methods Augmented with Ultrasonic Haptic Feedback. Proc. ACM Hum.-Comput. Interact. 6, 207–225 (2022).
[3] Cha, Y. & Myung, R. Extended Fitts' law for 3D pointing tasks using 3D target arrangements. Int. J. Ind. Ergon. 43, 350–355 (2013).
[4] Fernandes, A. S., Murdison, T. S. & Proulx, M. J. Leveling the Playing Field: A Comparative Reevaluation of Unmodified Eye Tracking as an Input and Interaction Modality for VR. IEEE Trans. Vis. Comput. Graph. 29, 2269–2279 (2023).
[5] Kytö, M., Ens, B., Piumsomboon, T., Lee, G. A. & Billinghurst, M. Pinpointing. Proc 2018 Chi Conf Hum Factors Comput Syst 1–14 (2018) doi:10.1145/3173574.3173655.
Whirling Interface
Base Idea: Motion Matching based Selection
Orbits [UIST`15]
[Image]
  • Gaze-based selection
  • Small targets
TraceMatch [Ubicomp`16]
[Image]
  • Hand gesture with computer vision technique
  • Explore applicability with different settings
One-handed Input [IMWUT`22]
[Image]
  • Touch screen gesture
  • Precise modality make input reliable

Showed potential of motion maching
>> Use reliable modaility(Hand motion) can be a way for small target selection on AR glasses.

There are still areas for improvement

  • Missing interaction features: feedback, confirmation, and cancellation
  • Missing comparison study: Is it better than ray casting?
    • User? System?
[1] Esteves, A., Velloso, E., Bulling, A. & Gellersen, H. Orbits: Gaze Interaction for Smart Watches using Smooth Pursuit Eye Movements. in Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology 457–466 (ACM, Charlotte NC USA, 2015). doi:10.1145/2807442.2807499.
[2] Lukowicz, P. et al. TraceMatch. Proc. 2016 ACM Int. Jt. Conf. Pervasive Ubiquitous Comput. 298–303 (2016) doi:10.1145/2971648.2971714.
[3] Esteves, A., Bouquet, E., Pfeuffer, K. & Alt, F. One-handed Input for Mobile Devices via Motion Matching and Orbits Controls. Proc Acm Interact Mob Wearable Ubiquitous Technologies 6, 1–24 (2022).
Whirling Interface
Improvements

Design Aligned with Don Norman's principles [1]

  • Real-time feedback to guide users
  • Intuitive confirmation and cancellation features

Techniques

  • Adaptive time window to improve responsiveness
  • Add input states to enable feedback/confirmation
    State transition with Corr.Coeff/RMS threshold
[Image]
Adaptive time window
[Image]
Input states
[1] Norman, D. A. Natural user interfaces are not natural. Interactions 17, 6–10 (2010).
Whirling Interface
How it works: Live Demo
  • Built with Mediapipe[1],Aframe[2](WebXR)
    -Recommend to test on laptop/PC
    -for mobile, landscape mode is recommended
    -can move your view with "WASD" or "Arrow keys"

  • Demo 1: 2 targets, very close

    [link to try on web]

  • Demo 2: 12 targets on 2m

    [link to try on web]
[1] Google. "MediaPipe." Google GitHub, 2023, https://github.com/google/mediapipe.
[2] A-Frame. "A-Frame – Make WebVR." A-Frame, 2023, https://aframe.io/.
Experiment
Experiment Setup

Participants

  • 16 right-handed
    Age 18-28 (M=23.4, SD=3.2)
    9 Male, 7 Female

Apparatus

  • Device: MS HoloLens 2
    MRTK for hand tracking/gesture
    used 'pinch' as confirmation gesture[1]
  • Target conditions
    • Placement: 12 targets in 4 x 3 grid
    • Size: 48mm(diameter)
      about 2.75° at 1m, 1.38° at 2m
  • Whirling conditions: 12 setttings
    • Speed(2)*: 180°/s, 240°/s
                  2 sec, 1.5 sec per cycle
    • Direction(2)*: Clockwise, CCW
    • Phase(3): 0°, 120°, 240°

Study Design

  • Within-subject design: 2 x 2
    • 2 distances (1m, 2m)
    • 2 methods (Whirling, Ray Casting)
  • Analysis
    RM-ANOVA
    Post-hoc: paired permutation test

Procedure

  • 4 sessions:
    2 distances × 2 methods
    Latin square balanced order
  • Trials: 48 per session
    • 4 repetitions for each placement
      including all spped/direction*
    • 15-second time limit per trial
      mark as failure if not completed
  • NASA-TLX after each session
  • Exp. time: 50 min/participant
[Image]
Near: 1 meter
[Image]
Far: 2 meters
Image captured from HoloLens 2 (FOV: 64.69°).
[1] Tafadzwa Joseph Dube, Yuan Ren, Hannah Limerick, I. Scott MacKenzie, and Ahmed Sabbir Arif. 2022. Push, Tap, Dwell, and Pinch: Evaluation of Four Mid-air Selection Methods Augmented with Ultrasonic Haptic Feedback. Proc. ACM Hum.-Comput. Interact. 6, ISS, Article 565 (December 2022), 19 pages. https://doi.org/10.1145/3567718
Experiment
Results

Success rate

  • Ray castingFar
    is worse
    Other conditions were comparable
[Image]

Completion time

  • 'Whirling' shows consistent performance across distances
  • Ray castingNear

    >
    WhirlingNear
    WhirlingFar

    >
    Ray castingFar
[Image]

NASA-TLXRaw TLX

  • Ray castingFar
    is worse
    Other conditions were comparable
[Image]
Experiment
Results: by Placements

Success rate

[Image]

Completion time

[Image]

Whirling showed consistent performance across placements
Ray casting exhibited lower performance, especially for corner/bottom targets

Experiment
Results: Sensing FOV

Sensing FoV

  • Whirling requires smaller sensing FOV than Ray casting
[Image]
Sensing FOV of the ray casting and Whirling method.

Whirling hand position

  • Hand position remained consistent regardless of target distance
[Image]
Distribution of hand positions during Whirling per participant.
Discussion
Findings

Importance of Input States

Whirling has better interactivity

  • Feedback enhance interactivity
    ''we know we are doing well while using Whirling, but for Ray casting, we don't know what is problem''
  • Cancellation feature crucial
    18% of trials needed multiple attempts
    • >>Including the cancellation...
  • Whirling was faster than Ray casting for distant targets
  • Whirling required less taskload than Ray casting
  • Users prefer Whirling over Ray casting in general
    10 out of 16 participants

Advantages of Sensing

Consistent performance
    across distances and placements

  • Smaller sensing field of view
  • Avoids self-occlusion issues and robust to minor tracking inconsistencies
    usually occur for Ray casting confirmation gesture
  • Potential for less accurate sensors or outdoor use
    use with IMU sensors or radar
Discussion
Applications & Future Work

Applications

  • Integration with AR/VR interfaces
    Enable interaction with significantly smaller buttons
  • Alternative selection method for AR glasses in realworld
    e.g., button on objects, input while walking
  • Potential for lightweigth glasses with only single RGB camera
    e.g., AI glasses
[Image]
Whirling Interface concept in a museum.

Limitations & Future Work

  • Need for comparative studies with other techniques
    • Ray casting with intention-aware features
    • Gaze-based selection
  • Address scalability issues
    • More diverse target speeds
    • Maximum target numbers in viewfield
    • Model difficulty index (e.g., Fitts' law of Whirling)
  • Lack of testing in diverse scenarios
    • Mobile scenarios
    • Selection as secondary task
    • Occupied hands
  • Improve detection with machine learning techniques
    • Feature based input state transition
    • Siamese network to replace correlation filter
  • Testing with other sensors
    • smartwatch IMU to support FoV free input
    • EMG, Radar, and others
Conclusion
Whirling Interface
Whirling Concept
  • Goal: Small target selection for AR/VR glasses
  • Whirling Interface
    • Better Interactivity
    • Consistent performance
  • Key Findings:
    • Better than Ray casting on small and distant targets
    • More consistent performance across distances and placements
    • Requires smaller field of view
  • Future Work:
    • Extend with scalability
    • Explore mobile scenarios
      (walking, talking, multitasking)
    • Evaluate with different sensors
      (single RGB camera, smartwatch(IMU), EMG, radar)
Funded by
  • IITP grant funded by the Korea government (MSIT) (No.RS-2024-00397663, Real-time XR Interface Technology Development for Environmental Adaptation)
  • Korea Institute for Advancement of Technology (KIAT) grant funded by the Korea Government (MOTIE) (P0012746, The Competency Development Program for Industry Specialist)
Hidden Slide
  • Example
Whirling Interface
Hand-based Motion Matching Selection
for Small Target on XR Displays
Juyoung Lee1, Seo Young Oh1, Minju Baeck1, Hui-Shyong Yeo2, Hyung-il Kim3,
Thad Starner4, Woontack Woo1,3
1KAIST UVR Lab.   2Huawei   3KI-ITC ARRC   4Georgia Tech
Scan for more info and demos
QR Code
https://juyounglee.net/projects/whirling
Looking for
collaboration or internship
Topics of interest:
  • AR for daily activities
  • Interaction for AR/AI glasses
  • topics related to AR/wearables