Operation System for Simulation Roadheader Based on Visual Motion Capture

Authors

Yongling Li, School of Mechanical and Electrical Engineering, Anhui Jianzhu University, Hefei 236001, China; Key Laboratory of Intelligent Manufacturing of Construction Machinery, Anhui Education Department, Hefei 230601, China; Sichuan Provincial Key Laboratory of Process Equipment and Control Engineering, Zigong 643000, China
Lingzhi Liu, School of Mechanical and Electrical Engineering, Anhui Jianzhu University, Hefei 236001, China; Key Laboratory of Intelligent Manufacturing of Construction Machinery, Anhui Education Department, Hefei 230601, China
Baishun Zhou, School of Computer Science, China University of Labor Relations, Beijing 100048, China
Jingfa Lei, School of Mechanical and Electrical Engineering, Anhui Jianzhu University, Hefei 236001, China; Key Laboratory of Intelligent Manufacturing of Construction Machinery, Anhui Education Department, Hefei 230601, China; Sichuan Provincial Key Laboratory of Process Equipment and Control Engineering, Zigong 643000, China
Miao Zhang, School of Mechanical and Electrical Engineering, Anhui Jianzhu University, Hefei 236001, China; Key Laboratory of Intelligent Manufacturing of Construction Machinery, Anhui Education Department, Hefei 230601, China
Ruhai Zhao, School of Mechanical and Electrical Engineering, Anhui Jianzhu University, Hefei 236001, China; Key Laboratory of Intelligent Manufacturing of Construction Machinery, Anhui Education Department, Hefei 230601, China

Abstract

Abstract: To enhance the natural human-machine interaction in simulation roadheader environment, a vision-based simulation roadheader operation system is proposed. The visual motion capture unit is based on the MediaPipe framework, which captures hand gestures through cameras and creates a correspondence between the physical world and virtual space. An improved Kalman filter algorithm is proposed by setting a weighted centroid to address the issue of unreasonable jumps in hand keypoint data during large-scale movements. The operator's gestures are discerned and the corresponding commands are conveyed. The results show that the improved method has significant advantages over the control group in terms of mean square error, signal-to-noise ratio, and approximate entropy parameters. The gesture recognition system is developed with an accuracy rate exceeding 92%. This interface enables the operator to efficiently control the simulated tunneling machine.

Recommended Citation

Li, Yongling; Liu, Lingzhi; Zhou, Baishun; Lei, Jingfa; Zhang, Miao; and Zhao, Ruhai (2025) "Operation System for Simulation Roadheader Based on Visual Motion Capture," Journal of System Simulation: Vol. 37: Iss. 6, Article 17.
DOI: 10.16182/j.issn1004731x.joss.24-0516
Available at: https://dc-china-simulation.researchcommons.org/journal/vol37/iss6/17

First Page

1531

Last Page

1541

CLC

TP391.9

Recommended Citation

Li Yongling, Liu Lingzhi, Zhou Baishun, et al. Operation System for Simulation Roadheader Based on Visual Motion Capture[J]. Journal of System Simulation, 2025, 37(6): 1531-1541.

Corresponding Author

Zhou Baishun

DOI

10.16182/j.issn1004731x.joss.24-0516