Whirling Interface
Hand-based Motion Matching Selection
for Small Target on XR Displays
Hand-based Motion Matching Selection
for Small Target on XR Displays
Juyoung Lee1, Seo Young Oh1, Minju Baeck1, Hui-Shyong Yeo2, Hyung-il Kim3,
Thad Starner4, Woontack Woo1,3
1KAIST UVR Lab. 2Huawei 3KI-ITC ARRC 4Georgia Tech
Thad Starner4, Woontack Woo1,3
1KAIST UVR Lab. 2Huawei 3KI-ITC ARRC 4Georgia Tech
Contents
Whirling Interface
- Goal:
Develop a target selection interface suitable for daily use - Whirling Interface
- Improved Interactivity
- Consistent performance
- Experiment
- Ray casting vs Whirling
- 48mm targets in {1m, 2m} distance
- Discussion
- Findings
- Applications
- Limitations
PS2 – 3DUIs 1
Whirling Interface
Juyoung Lee, 2025-04-01
Introduction
Smartglasses is here
From Immersive headset with Spatial computing
to lightweight smartglasses to use in dailylife
MS HoloLens 2
Apple Vision Pro
Magic Leap 2
Vuzix Z100
Inmo Go2
Vuzix Shield
Even Realities G1
PS2 – 3DUIs 1
Whirling Interface
Juyoung Lee, 2025-04-01
Introduction
Goal
Create an interface for daily use AR
with small targets
and bare-hand interaction
Why 'small targets'?
- Minimize obstruction of the real world
- Accommodate display's limited Field of View (FOV) in AR glasses
Why 'bare-hand' interaction?
- Hand is the most common input modalityEven the very early AR research suggested it[1]
- No need to carry controllers all the time
- Enables input even when hands are occupied by other objects
[1] Thad Starner, Steve Mann, Bradley Rhodes, Jeffrey Levine, Jennifer Healey, Dana Kirsch, Rosalind W. Picard, Alex Pentland; Augmented Reality through Wearable Computing. Presence: Teleoperators and Virtual Environments 1997; 6 (4): 386–398. doi: https://doi.org/10.1162/pres.1997.6.4.386
PS2 – 3DUIs 1
Whirling Interface
Juyoung Lee, 2025-04-01
Related Work
Selection Methods for AR/VR
Ray Casting
Most common method
![[Image]](/static/ismar24/figures/extended_fittslaw.jpg)
Extended Fitts' law for 3D [3]
- Challenges:
- Dependent on tracking[1]
Tracking loss by out of FoV, Occlusion, Jitter, and others - Requires confirmation gesture
Gesture affects performance[2] - Small targets are hard to select
Following Fitts' Law[3]
Eye gaze pointing
Intuitive and ease of use
Photo: Apple, Vision Pro
- Challenges:
- Dependent on tracking
Higher miss rate without pointer[4] - Requires confirmation gesture
- Tracking issues on outdoors with sunlight
Multi-Modal/Step Input
Can complement methods
![[Image]](/static/ismar24/figures/pinpointing.png)
Pinpointing [5]
- Challenges:
- Requires more attention to use multi-modal
- Have to remember multiple steps/modal combinations
- Precise but slower
[1] Babu, S. V., Huang, H.-C., Teather, R. J. & Chuang, J.-H. Comparing the Fidelity of Contemporary Pointing with Controller Interactions on Performance of Personal Space Target Selection. 2022 IEEE Int. Symp. Mix. Augment. Real. (ISMAR) 00, 404–413 (2022).
[2] Dube, T. J., Ren, Y., Limerick, H., MacKenzie, I. S. & Arif, A. S. Push, Tap, Dwell, and Pinch: Evaluation of Four Mid-air Selection Methods Augmented with Ultrasonic Haptic Feedback. Proc. ACM Hum.-Comput. Interact. 6, 207–225 (2022).
[3] Cha, Y. & Myung, R. Extended Fitts' law for 3D pointing tasks using 3D target arrangements. Int. J. Ind. Ergon. 43, 350–355 (2013).
[4] Fernandes, A. S., Murdison, T. S. & Proulx, M. J. Leveling the Playing Field: A Comparative Reevaluation of Unmodified Eye Tracking as an Input and Interaction Modality for VR. IEEE Trans. Vis. Comput. Graph. 29, 2269–2279 (2023).
[5] Kytö, M., Ens, B., Piumsomboon, T., Lee, G. A. & Billinghurst, M. Pinpointing. Proc 2018 Chi Conf Hum Factors Comput Syst 1–14 (2018) doi:10.1145/3173574.3173655.
[2] Dube, T. J., Ren, Y., Limerick, H., MacKenzie, I. S. & Arif, A. S. Push, Tap, Dwell, and Pinch: Evaluation of Four Mid-air Selection Methods Augmented with Ultrasonic Haptic Feedback. Proc. ACM Hum.-Comput. Interact. 6, 207–225 (2022).
[3] Cha, Y. & Myung, R. Extended Fitts' law for 3D pointing tasks using 3D target arrangements. Int. J. Ind. Ergon. 43, 350–355 (2013).
[4] Fernandes, A. S., Murdison, T. S. & Proulx, M. J. Leveling the Playing Field: A Comparative Reevaluation of Unmodified Eye Tracking as an Input and Interaction Modality for VR. IEEE Trans. Vis. Comput. Graph. 29, 2269–2279 (2023).
[5] Kytö, M., Ens, B., Piumsomboon, T., Lee, G. A. & Billinghurst, M. Pinpointing. Proc 2018 Chi Conf Hum Factors Comput Syst 1–14 (2018) doi:10.1145/3173574.3173655.
PS2 – 3DUIs 1
Whirling Interface
Juyoung Lee, 2025-04-01
Whirling Interface
Base Idea: Motion Matching based Selection
Orbits [UIST`15]
![[Image]](/static/ismar24/figures/orbits_2015.png)
- Gaze-based selection
- Small targets
TraceMatch [Ubicomp`16]
![[Image]](/static/ismar24/figures/tracematch_ubicomp16.png)
- Hand gesture with computer vision technique
- Explore applicability with different settings
One-handed Input [IMWUT`22]
![[Image]](/static/ismar24/figures/onehanded_imwut22.png)
- Touch screen gesture
- Precise modality make input reliable
Showed potential of motion maching
>> Use reliable modaility(Hand motion) can be a way for small target selection on AR glasses.
There are still areas for improvement
- Missing interaction features: feedback, confirmation, and cancellation
- Missing comparison study: Is it better than ray casting?
- User? System?
[1] Esteves, A., Velloso, E., Bulling, A. & Gellersen, H. Orbits: Gaze Interaction for Smart Watches using Smooth Pursuit Eye Movements. in Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology 457–466 (ACM, Charlotte NC USA, 2015). doi:10.1145/2807442.2807499.
[2] Lukowicz, P. et al. TraceMatch. Proc. 2016 ACM Int. Jt. Conf. Pervasive Ubiquitous Comput. 298–303 (2016) doi:10.1145/2971648.2971714.
[3] Esteves, A., Bouquet, E., Pfeuffer, K. & Alt, F. One-handed Input for Mobile Devices via Motion Matching and Orbits Controls. Proc Acm Interact Mob Wearable Ubiquitous Technologies 6, 1–24 (2022).
[2] Lukowicz, P. et al. TraceMatch. Proc. 2016 ACM Int. Jt. Conf. Pervasive Ubiquitous Comput. 298–303 (2016) doi:10.1145/2971648.2971714.
[3] Esteves, A., Bouquet, E., Pfeuffer, K. & Alt, F. One-handed Input for Mobile Devices via Motion Matching and Orbits Controls. Proc Acm Interact Mob Wearable Ubiquitous Technologies 6, 1–24 (2022).
PS2 – 3DUIs 1
Whirling Interface
Juyoung Lee, 2025-04-01
Whirling Interface
Improvements
Design Aligned with Don Norman's principles [1]
- Real-time feedback to guide users
- Intuitive confirmation and cancellation features
Techniques
- Adaptive time window to improve responsiveness
- Add input states to enable feedback/confirmation
State transition with Corr.Coeff/RMS threshold
![[Image]](/static/ismar24/figures/adaptive_window.png)
Adaptive time window
![[Image]](/static/ismar24/figures/whirling_inputstate.png)
Input states
[1] Norman, D. A. Natural user interfaces are not natural. Interactions 17, 6–10 (2010).
PS2 – 3DUIs 1
Whirling Interface
Juyoung Lee, 2025-04-01
Whirling Interface
How it works: Live Demo
- Built with Mediapipe[1],Aframe[2](WebXR)
-Recommend to test on laptop/PC
-for mobile, landscape mode is recommended
-can move your view with "WASD" or "Arrow keys" -
Demo 1: 2 targets, very close
[link to try on web] Demo 2: 12 targets on 2m
[link to try on web]
[1] Google. "MediaPipe." Google GitHub, 2023, https://github.com/google/mediapipe.
[2] A-Frame. "A-Frame – Make WebVR." A-Frame, 2023, https://aframe.io/.
[2] A-Frame. "A-Frame – Make WebVR." A-Frame, 2023, https://aframe.io/.
PS2 – 3DUIs 1
Whirling Interface
Juyoung Lee, 2025-04-01
Experiment
Experiment Setup
Participants
- 16 right-handed
Age 18-28 (M=23.4, SD=3.2)
9 Male, 7 Female
Apparatus
- Device: MS HoloLens 2
MRTK for hand tracking/gesture
used 'pinch' as confirmation gesture[1] - Target conditions
- Placement: 12 targets in 4 x 3 grid
- Size: 48mm(diameter)about 2.75° at 1m, 1.38° at 2m
- Whirling conditions: 12 setttings
- Speed(2)*: 180°/s, 240°/s
2 sec, 1.5 sec per cycle - Direction(2)*: Clockwise, CCW
- Phase(3): 0°, 120°, 240°
- Speed(2)*: 180°/s, 240°/s
Study Design
- Within-subject design: 2 x 2
- 2 distances (1m, 2m)
- 2 methods (Whirling, Ray Casting)
- Analysis
RM-ANOVA
Post-hoc: paired permutation test
Procedure
- 4 sessions:
2 distances × 2 methods
Latin square balanced order - Trials: 48 per session
- 4 repetitions for each placement
including all spped/direction* - 15-second time limit per trial
mark as failure if not completed
- 4 repetitions for each placement
- NASA-TLX after each session
- Exp. time: 50 min/participant
![[Image]](/static/ismar24/figures/realview_near_ray.jpg)
Near: 1 meter
![[Image]](/static/ismar24/figures/realview_far_whirling.jpg)
Far: 2 meters
Image captured from HoloLens 2 (FOV: 64.69°).
[1] Tafadzwa Joseph Dube, Yuan Ren, Hannah Limerick, I. Scott MacKenzie, and Ahmed Sabbir Arif. 2022. Push, Tap, Dwell, and Pinch: Evaluation of Four Mid-air Selection Methods Augmented with Ultrasonic Haptic Feedback. Proc. ACM Hum.-Comput. Interact. 6, ISS, Article 565 (December 2022), 19 pages. https://doi.org/10.1145/3567718
PS2 – 3DUIs 1
Whirling Interface
Juyoung Lee, 2025-04-01
Experiment
Results
Success rate
- Ray castingFaris worse
Other conditions were comparable
![[Image]](/static/ismar24/figures/Accuracy.png)
Completion time
- 'Whirling' shows consistent performance across distances
-
Ray castingNear
>WhirlingNearWhirlingFar
>Ray castingFar
![[Image]](/static/ismar24/figures/Completion%20Time(sec).png)
NASA-TLXRaw TLX
- Ray castingFaris worse
Other conditions were comparable
![[Image]](/static/ismar24/figures/tlx_overall.png)
PS2 – 3DUIs 1
Whirling Interface
Juyoung Lee, 2025-04-01
Experiment
Results: by Placements
Success rate
![[Image]](/static/ismar24/figures/accuracy_by_placements_sep.png)
Completion time
![[Image]](/static/ismar24/figures/time_by_placements_sep.png)
Whirling showed consistent performance across placements
Ray casting exhibited lower performance, especially for corner/bottom targets
PS2 – 3DUIs 1
Whirling Interface
Juyoung Lee, 2025-04-01
Experiment
Results: Sensing FOV
Sensing FoV
- Whirling requires smaller sensing FOV than Ray casting
![[Image]](/static/ismar24/figures/fov_ray_orbit_new.png)
Sensing FOV of the ray casting and Whirling method.
Whirling hand position
- Hand position remained consistent regardless of target distance
![[Image]](/static/ismar24/figures/position_distribution.png)
Distribution of hand positions during Whirling per participant.
PS2 – 3DUIs 1
Whirling Interface
Juyoung Lee, 2025-04-01
Discussion
Findings
Importance of Input States
Whirling has better interactivity
- Feedback enhance interactivity
''we know we are doing well while using Whirling, but for Ray casting, we don't know what is problem'' - Cancellation feature crucial
18% of trials needed multiple attempts - Whirling was faster than Ray casting for distant targets
- Whirling required less taskload than Ray casting
- Users prefer Whirling over Ray casting in general
10 out of 16 participants
>>Including the cancellation...
Advantages of Sensing
Consistent performance
across distances and placements
-
Smaller sensing field of view
- Avoids self-occlusion issues and robust to minor tracking inconsistencies
usually occur for Ray casting confirmation gesture - Potential for less accurate sensors or outdoor use
use with IMU sensors or radar
PS2 – 3DUIs 1
Whirling Interface
Juyoung Lee, 2025-04-01
Discussion
Applications & Future Work
Applications
- Integration with AR/VR interfaces
Enable interaction with significantly smaller buttons - Alternative selection method for AR glasses in realworld
e.g., button on objects, input while walking - Potential for lightweigth glasses with only single RGB camera
e.g., AI glasses
![[Image]](/static/ismar24/figures/museum_application.png)
Whirling Interface concept in a museum.
Limitations & Future Work
- Need for comparative studies with other techniques
- Ray casting with intention-aware features
- Gaze-based selection
- Address scalability issues
- More diverse target speeds
- Maximum target numbers in viewfield
- Model difficulty index (e.g., Fitts' law of Whirling)
- Lack of testing in diverse scenarios
- Mobile scenarios
- Selection as secondary task
- Occupied hands
- Improve detection with machine learning techniques
- Feature based input state transition
- Siamese network to replace correlation filter
- Testing with other sensors
- smartwatch IMU to support FoV free input
- EMG, Radar, and others
PS2 – 3DUIs 1
Whirling Interface
Juyoung Lee, 2025-04-01
Conclusion
Whirling Interface
- Goal: Small target selection for AR/VR glasses
- Whirling Interface
- Better Interactivity
- Consistent performance
- Key Findings:
- Better than Ray casting on small and distant targets
- More consistent performance across distances and placements
- Requires smaller field of view
- Future Work:
- Extend with scalability
- Explore mobile scenarios(walking, talking, multitasking)
- Evaluate with different sensors(single RGB camera, smartwatch(IMU), EMG, radar)
Funded by
- IITP grant funded by the Korea government (MSIT) (No.RS-2024-00397663, Real-time XR Interface Technology Development for Environmental Adaptation)
- Korea Institute for Advancement of Technology (KIAT) grant funded by the Korea Government (MOTIE) (P0012746, The Competency Development Program for Industry Specialist)
PS2 – 3DUIs 1
Whirling Interface
Juyoung Lee, 2025-04-01
Whirling Interface
Hand-based Motion Matching Selection
for Small Target on XR Displays
Hand-based Motion Matching Selection
for Small Target on XR Displays
Juyoung Lee1, Seo Young Oh1, Minju Baeck1, Hui-Shyong Yeo2, Hyung-il Kim3,
Thad Starner4, Woontack Woo1,3
1KAIST UVR Lab. 2Huawei 3KI-ITC ARRC 4Georgia Tech
Thad Starner4, Woontack Woo1,3
1KAIST UVR Lab. 2Huawei 3KI-ITC ARRC 4Georgia Tech
Scan for more info and demos
https://juyounglee.net/projects/whirling
Looking for
collaboration or internship
collaboration or internship
Topics of interest:
- AR for daily activities
- Interaction for AR/AI glasses
- topics related to AR/wearables
Whirling Interface Hand-based Motion Matching Selection for Small Target on XR Displays Juyoung Lee 1 , Seo Young Oh 1 , Minju Baeck 1 , Hui-Shyong Yeo 2 , Hyung-il Kim 3 , Thad Starner 4 , Woontack Woo 1,3 1 KAIST UVR Lab. 2 Huawei 3 KI-ITC ARRC 4 Georgia Tech this is collobaration work from three different country: KAIST, Huawei, GeorgiaTech