AR-assisted Sign Language Letter Recognition Method Based on Improved MobileNet Network

Authors

Chunhong Liu, School of Computer Science & Technology, Southwest University of Science and Technology, Mianyang 621010, ChinaFollow
Song Wang, School of Computer Science & Technology, Southwest University of Science and Technology, Mianyang 621010, ChinaFollow
Fupan Wang, School of Computer Science & Technology, Southwest University of Science and Technology, Mianyang 621010, China
Wensheng Tang, School of Computer Science & Technology, Southwest University of Science and Technology, Mianyang 621010, China
Yunqiang Pei, School of Computer Science & Technology, Southwest University of Science and Technology, Mianyang 621010, China
Dongsheng Tian, School of Computer Science & Technology, Southwest University of Science and Technology, Mianyang 621010, China
Yadong Wu, School of Computer Science & Engineering, Sichuan University of Science and Engineering, Zigong 643002, China

Abstract

An AR-assisted sign language letter recognition algorithm MS-MobileNet is proposed for the problems of sign language gestures needing to be standardized and low recognition rate. A multi-scale convolution module is designed to extract the low-level features and enhance the feature extraction ability. ELU activation function is used to retain the negative feature information, which combined with a lightweight MobileNet model for the web to improve the recognition accuracy and real-time performance for mobile AR applications. Test results show that compared with the original model, the recognition accuracy of MS-MobileNet on the datasets ASL-M, NUS-II and Creative Senz3D is improved by 2.58%, 5.32% and 3.04%, respectively. Based on improved network, a WebAR-assisted sign language collaborative interaction system is designed. After the evaluation test, the average user participation rate reached 8.2 points, and the single recognition time is less than 0.115 s. User's needs for immersive realtime sign language letter interaction is better met.

Recommended Citation

Liu, Chunhong; Wang, Song; Wang, Fupan; Tang, Wensheng; Pei, Yunqiang; Tian, Dongsheng; and Wu, Yadong (2023) "AR-assisted Sign Language Letter Recognition Method Based on Improved MobileNet Network," Journal of System Simulation: Vol. 35: Iss. 6, Article 15.
DOI: 10.16182/j.issn1004731x.joss.22-0216
Available at: https://dc-china-simulation.researchcommons.org/journal/vol35/iss6/15

First Page

1308

Revised Date

2022-05-25

DOI Link

http://dx.doi.org/10.16182/j.issn1004731x.joss.22-0216

Last Page

1321

CLC

TP391

Recommended Citation

Liu Chunhong, Wang Song, Wang Fupan, et al. AR-assisted Sign Language Letter Recognition Method Based on Improved MobileNet Network[J]. Journal of System Simulation, 2023, 35(6): 1308-1321.

DOI

10.16182/j.issn1004731x.joss.22-0216