Extended Reality (XR) Interfaces for Unmanned System Control: A Systematic Review

Sangwoo Jeon, Raguvaran S.

Abstract


Extended reality technologies—encompassing virtual reality (VR), augmented reality (AR), and mixed reality (MR)—offer transformative potential for unmanned system control by providing intuitive spatial interfaces, immersive situational awareness, and natural interaction paradigms that transcend traditional screen-based teleoperation. This systematic review comprehensively examines XR interface designs, interaction techniques, and empirical evaluations across unmanned aerial, ground, marine, and space systems. We systematically categorize XR interfaces by their reality-virtuality continuum position: fully immersive VR environments reconstructing remote operational spaces from sensor data, AR overlays augmenting operator’s physical environment with telemetry and video feeds, and MR interfaces blending physical props with virtual elements. The review analyzes visualization approaches including ego-centric perspectives placing operators virtually aboard vehicles, exo-centric views providing external vantage points, and 3D spatial data representation through point cloud rendering and volumetric reconstruction. We examine interaction paradigms leveraging XR capabilities including gaze-based selection, gesture recognition, voice control, and haptic feedback through controllers or wearables. Empirical studies are synthesized showing 15-35% task completion time improvements for spatial tasks, enhanced situational awareness, but mixed workload results. Application-specific designs are analyzed for UAV swarm control, underwater ROV teleoperation, planetary rover operation, and surgical robotics. Technical challenges are critically evaluated including latency requirements below 20ms, field-of-view limitations, and computational demands for real-time 3D reconstruction. Human factors considerations are examined including cybersickness, visual fatigue, and safety concerns from reduced physical awareness. The review identifies critical research gaps including lack of standardized evaluation protocols, insufficient long-term studies, and limited investigation of XR interfaces under degraded conditions.

Keywords


extended reality, virtual reality, augmented reality, human-machine interfaces.

References


Ha, Changsu, and Dongjun Lee. “Vision-Based Teleoperation of Unmanned Aerial and Ground Vehicles”. In 2013 IEEE International Conference on Robotics and Automation, 1465–70, 2013. https://doi.org/10.1109/ICRA.2013.6630764.

Kalinov, Ivan, Daria Trinitatova, and Dzmitry Tsetserukou. “WareVR: Virtual Reality Interface for Supervision of Autonomous Robotic System Aimed at Warehouse Stocktaking”. In 2021 IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2139–45, 2021. https://doi.org/10.1109/SMC52423.2021.9659133.

Lee, Jongseok, Ribin Balachandran, Yuri S. Sarkisov, Marco De Stefano, Andre Coelho, Kashmira Shinde, Min Jun Kim, Rudolph Triebel, and Konstantin Kondak. “Visual-Inertial Telepresence for Aerial Manipulation”. In 2020 IEEE International Conference on Robotics and Automation (ICRA), 1222–29, 2020. https://doi.org/10.1109/ICRA40945.2020.9197394.

Macchini, Matteo, Manana Lortkipanidze, Fabrizio Schiano, and Dario Floreano. “The Impact of Virtual Reality and Viewpoints in Body Motion Based Drone Teleoperation”. In 2021 IEEE Virtual Reality and 3D User Interfaces (VR), 511–18, 2021. https://doi.org/10.1109/VR50410.2021.00075.

Makhataeva, Zhanat, and Huseyin Atakan Varol. “Augmented Reality for Robotics: A Review”. Robotics 9, no. 2 (2020). https://doi.org/10.3390/robotics9020021.

Naceri, Abdeldjallil, Dario Mazzanti, Joao Bimbo, Yonas T. Tefera, Domenico Prattichizzo, Darwin G. Caldwell, Leonardo S. Mattos, and Nikhil Deshpande. “The Vicarios Virtual Reality Interface for Remote Robotic Teleoperation”. Journal of Intelligent & Robotic Systems 101, no. 4 (3 April 2021): 80. https://doi.org/10.1007/s10846-021-01311-7.

Pan, Yong, Chengjun Chen, Dongnian Li, Zhengxu Zhao, and Jun Hong. “Augmented Reality-Based Robot Teleoperation System Using RGB-D Imaging and Attitude Teaching Device”. Robotics and Computer-Integrated Manufacturing 71 (2021): 102167. https://doi.org/10.1016/j.rcim.2021.102167.

Ponomareva, Polina, Daria Trinitatova, Aleksey Fedoseev, Ivan Kalinov, and Dzmitry Tsetserukou. “GraspLook: A VR-Based Telemanipulation System with R-CNN-Driven Augmentation of Virtual Environment”. In 2021 20th International Conference on Advanced Robotics (ICAR), 166–71, 2021. https://doi.org/10.1109/ICAR53236.2021.9659460.

Spano, Lucio Davide. “Teleoperating Humanoids Robots Using Standard VR Headsets: A Systematic Review”. In Proceedings of the 5th International Conference on Computer-Human Interaction Research and Applications - Volume 1: Humanoid, 308–16. INSTICC, 2021. https://doi.org/10.5220/0010723200003060.

Stotko, Patrick, Stefan Krumpen, Max Schwarz, Christian Lenz, Sven Behnke, Reinhard Klein, and Michael Weinmann. “A VR System for Immersive Teleoperation and Live Exploration with a Mobile Robot”. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 3630–37, 2019. https://doi.org/10.1109/IROS40897.2019.8968598.

Walker, Michael E., Hooman Hedayati, and Daniel Szafir. “Robot Teleoperation with Augmented Reality Virtual Surrogates”. In 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI), 202–10, 2019. https://doi.org/10.1109/HRI.2019.8673306.

Yashin, Grigoriy A., Daria Trinitatova, Ruslan T. Agishev, Roman Ibrahimov, and Dzmitry Tsetserukou. “AeroVr: Virtual Reality-Based Teleoperation with Tactile Feedback for Aerial Manipulation”. In 2019 19th International Conference on Advanced Robotics (ICAR), 767–72, 2019. https://doi.org/10.1109/ICAR46387.2019.8981574.


Refbacks

  • There are currently no refbacks.


Bookmark and Share


Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 License.