1 COEP Technological University (COEP Tech), School of Engineering and Technology, Department of Manufacturing Engineering and Industrial Management, Pune, India
2 Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI), Robotics, Abu Dhabi, United Arab Emirates
3 Dr. Vishwanath Karad MIT World Peace University (MIT-WPU), Department of Computer Engineering and Technology, Computer Science and Engineering, Pune, India
4 SRH Berlin University of Applied Sciences, Department of Engineering and Sustainable Technology Management, Engineering and Sustainable Technology Management in Industry 4.0: Automation, Robotics & 3D Manufacturing, Berlin, Germany
Abstract
Traditional robotic teleoperation in industrial and hazardous environments often relies on physical joysticks, wearable sensor gloves, or computationally intensive depth-sensing cameras. These conventional methods inherently introduce significant deployment costs, mechanical constraints, and high computational overhead. To address these limitations, this paper presents the development and evaluation of a lightweight, low-cost monocular vision-based teleoperation framework designed specifically for a 4-degree-of-freedom (4-DOF) 3D-printed articulated robotic arm. Utilizing a standard consumer-grade RGB webcam, the proposed system extracts continuous hand skeletal keypoints in real time via the MediaPipe framework. To deliberately minimize processing overhead and eliminate the need for specialized GPU hardware, the algorithm isolates only the thumb tip (Landmark 4) and index fingertip (Landmark 8). A training-free heuristic threshold engine translates the calculated Euclidean distance between these two spatial coordinates into discrete servo commands, including Grip, Toggle, and Idle. Crucially, a software-level dynamic deadzone filter is integrated to explicitly reject sub-pixel landmark noise, successfully suppressing 85% of unwanted mechanical micro-motions during stationary poses. Hardware actuation is seamlessly executed via asynchronous PyFirmata serial communication to an Arduino Uno microcontroller. Rigorous experimental evaluation across 100,000 continuous inference cycles demonstrated a highly responsive mean end-to-end processing latency of 44.2 ms, comfortably satisfying the 100 ms upper bound for seamless Human-Robot Interaction (HRI). Furthermore, the system achieved a robust overall gesture classification F1-score of 95.2% across 5,472 controlled test cycles. Ultimately, this study validates that an optimized deterministic heuristic approach provides a highly efficient, cost-effective alternative to deep learning classifiers for real-time robotic teleoperation.
PAWAR , A. S., PATIL , S. M., SHAHANE , G. J., NIBE , N. V., PATIL , S. S., & MAHAJAN , A. M. (2026). Development of a Vision-Based Heuristic Control Framework for Multi-Axis Articulated Robot. MAS Journal of Applied Sciences, 11(2), 252–267. https://doi.org/10.5281/zenodo.20343082
📄Alamin, M.O., Khadir, E.M., Babiker, S.F., 2016. A vision-based teleoperation method for a robotic arm with 4 degrees of freedom. Conference of Basic Sciences and Engineering Studies (SGCAC), Conference Proceedings Book, 20-23 Februaty, Khartoum, Sudan, pp. 19-23.
📄Amprimo, G., Masi, G., Pettiti, G., Olmo, G., Priano, L., Ferraris, C., 2024. Hand tracking for clinical applications: Validation of the Google MediaPipe Hand (GMH) and the depth-enhanced GMH-D frameworks. Biomedical Signal Processing and Control, 96(A): 106508.
📄Chenchireddy, K., Dora, R., Mulla, G.B., Jegathesan, V., Sydu, S.A., 2024. Development of robotic arm control using arduino controller. IAES International Journal of Robotics and Automation, 13(3): 264–271.
📄Cui, C., Sunar, M.S., Su, G.E. 2025. Deep vision-based real-time hand gesture recognition: A review. PeerJ Computer Science, 11: p.e2921.
📄da Silva, M.V., Carvalho, M.H., Negri, J., Segreto Silva, T.H., Lahr, G.J.G., Godoy, R.V., Beckeret, M., 2025. A vision-based shared-control teleoperation scheme for controlling the robotic arm of a four-legged robot. Latin American Robotics Symposium (LARS), Symposium Proceedings Book, October 29-November 1 Monterrey, Nuevo León, Mexico: IEEE, pp. 1-6.
📄Drăgoi, M.V., Frimu, A.V., Postelnicu, A., Puiu, R.A., Petrea, G., Hank, A., 2026. Interactive teleoperation of an articulated robotic arm using vision-based human hand tracking. Biomimetics, 11(2):151.
📄Ibrahima, N.J., Ali, Y.H., Alzuabidi, I.A.F., Ahmed, A.A., Mohammed, B.S., Al-Dmoure, N.A., Abdeldayem, M., 2025. Hand gesture recognition for human-computer ınteraction using deep learning approach. Procedia Computer Science, 275: 876–884.
📄Jalayer, R., Jalayer, M., Orsenigo, C., Tomizuka, M., 2026. A review on deep learning for vision-based hand detection, hand segmentation and hand gesture recognition in human–robot interaction. Robotics and Computer-Integrated Manufacturing, 97: 103110.
📄Jarecki, A., Lee, K., 2024. Mixed reality-based teleoperation of mobile robotic arm: System apparatus and experimental case study. 21st International Conference on Ubiquitous Robots (UR), New York, NY, USA: IEEE, pp. 198–203.
📄Lopes, M.C., Silva, P.A., Marenco, L., Vilas Boas, E.C., Carvalho, J.G.A., Ferreira, C.A., Carvalho, A.L.O., Guimarães, C.V.R., Aquino, G.P., Figueiredo, F.A.P., 2026. GECO: A real-time computer vision-assisted gesture controller for advanced ıot home system. Sensors, 26(1):61.
📄Mohamed, N., Mustafa, M.B., Jomhari, N., 2021. A review of the hand gesture recognition system: current progress and future directions. IEEE Access, 9: 157422–157436.
📄Nuzzi, C., Pasinetti, S., Lancini, M., Docchio, F., Sansoni, G. 2018. Deep learning-based hand gesture recognition for collaborative robots. IEEE Instrumentation & Measurement Magazine, 22(2): 44-51.
📄Papanagiotou, D., Senteri, G., Manitsaris, S., 2021. Egocentric gesture recognition using 3d convolutional neural networks for the spatiotemporal adaptation of collaborative robots. Frontiers in Neurorobotics, 15: 703545.
📄Su, H., Qi, W., Chen, J., Yang, C., Sandoval, J., Laribi, M.A., 2023. Recent advancements in multimodal human–robot interaction. Frontiers in Neurorobotics, 17:1084000.
📄Wameed, M., Alkamachi, A.M., 2023a. Hand Gestures Robotic Control Based on Computer Vision. International Journal of Intelligent Systems and Applications in Engineering, 11(2): 1013.
📄Wameed, M., Alkamachi, A.M., Erçelebi, E., 2023b. Tracked Robot Control with Hand Gesture Based on MediaPipe. Al-Khwarizmi Engineering Journal, 19(3): 56–71.
📄Xavier, S., Vaisakh, B., Pai, M. L., 2023. Real-time hand gesture recognition using mediapipe and artificial neural networks. 14th International Conference on Computing Communication and Networking Technologies (ICCCNT), Conference Proceedings Book, 6-8 July, Delhi, India: IEEE, pp. 1-6.
📄Xie, J., Xu, Z., Zeng, J., Gao, Y., Hashimoto, K., 2025. Human–Robot ınteraction using dynamic hand gesture for teleoperation of quadruped robots with a robotic arm. Electronics, 14(5):860.
📄Zhang, G., Su, J., Zhang, S., Qi, J., Hou, Z., Lin, Q., 2025. Research on deep learning-based human–robot static/dynamic gesture-driven control framework. Sensors, 25(23):7203.