Software Sign Language Translator to Text and Speech by Using the Landmarks Technique of MediaPipe

Authors

  • Pattaranat Sriboonruang Division of Electronic and Telecommunication Engineering Faculty of Engineering Rajamangala University of Technology Krungthep
  • Pantakorn Tananchai Division of Electronic and Telecommunication Engineering Faculty of Engineering Rajamangala University of Technology Krungthep
  • Kachanthep Khaggathog Division of Electronic and Telecommunication Engineering Faculty of Engineering Rajamangala University of Technology Krungthep
  • Wuttichai Vijitkunsawat Division of Electronic and Telecommunication Engineering Faculty of Engineering Rajamangala University of Technology Krungthep
  • Pramote Anunvrapong Division of Electronic and Telecommunication Engineering Faculty of Engineering Rajamangala University of Technology Krungthep

Keywords:

Thai Sign Language, Sign Language Recognition, MediaPipe, Artificial Neural Network, LSTM algorithm

Abstract

Many deaf people and people with disabled hearing are commonly seen in the community. Especially in Thailand, there is the second leading disability type among all. This paper studies the performance of Thai Sign Language Recognition (TSLR) with the Long Short-Term Memory (LSTM) algorithm on the landmarks technique of the MediaPipe. We separate the three scenarios for our experiments: the Finger-Spelling for digit numbers (1-9), Natural Sign Language for 20 words, and Software Sign Language Translator (SSLT). The results show that Finger-Spelling and Natural Sign Language accuracy on the SSLT are between 94% - 99% and 80.5%, respectively.

References

กมลา ไกรฤกษ์, “หนังสือคู่มือภาษามือ” พิมพ์ครั้งที่ 1, กรุงเทพฯ: สำนักพิมพ์คุรุสภาลาดพร้าว, 2544.

A. Chaikaew, K. Somkuan and P. Sarapee, “Mobile Application for Thai Sign Language”, in 22nd International Computer Science and Engineering Conference (ICSEC), Chiang Mai, Thailand, 2018, pp. 1-4.

P. Nakjai and T. Katanyukul, “Hand sign recognition for thai finger-spelling: An application of convolution neural network”, Journal of Signal Processing Systems (2019), pp. 131-146.

A. Chaikaew, K. Sornkuan, and T. Yuyen, “Thai Sign Language Recognition: an Application of Deep Neural Network” in International ECTI Northern Section Conference on Electrical, Electronics, Computer and Telecommunications Engineering (ECTI-CON), Cha-am, Thailand, 2021, pp. 128-131.

A. Chaikeaw, “An applied holistic landmark with deep learning for Thai sign language recognition” in International Technical Conference on Circuits/Systems, Computers and Communication (ITC-CSCC), Phuket, Thailand, pp. 1046-1049.

F. Zhang, V. Bazaresky, A. Vakunov et al., “MediaPipe Hand: On-device Real-time Hand Tracking”, in CVPR Workshop on Computer Vision for Augmented and Virtual Reality (CVPR), Seattle, USA, 2020.

G. Anikumar, Fouzia M S, Anisha GS, “Imperative Methodology to Detech the Palm Gestures (American Sign Language) using YOLOv5 and MediaPipe” in 2nd International Conference on Intelligent Technology (CONIT), Hubli, India, 2022, pp. 1-4.

J. Brownlee, “Long Short-Term Memory Networks with Python”, Edition 1.0, Published eBook 2017.

R. Sangpal, T. Gawand, S. Vaykar and N. Madhavi, “JARVIS: An interpretation of AIML with integration of gTTS and Python”, 2nd International Conference on Intelligent Computing, Instrumentation and Control Technology (ICICICT), Kannur, India, 2019, pp. 486-489.

Downloads

Published

2022-12-29

How to Cite

Sriboonruang, P. ., Tananchai, P. ., Khaggathog, K. ., Vijitkunsawat, W. ., & Anunvrapong, P. (2022). Software Sign Language Translator to Text and Speech by Using the Landmarks Technique of MediaPipe . Journal of Industrial Technology : Suan Sunandha Rajabhat University, 10(2), 66–76. retrieved from https://ph01.tci-thaijo.org/index.php/fit-ssru/article/view/249506

Issue

Vol. 10 No. 2 (2022): ๋July-December 2022

Section

Research Articles

License

Copyright (c) 2022 Faculty of Industrial Technology, Suan Sunandha Rajabhat University

Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

บทความที่ได้รับการตีพิมพ์เป็นลิขสิทธิ์ของคณะเทคโนโลยีอุตสาหกรรม มหาวิทยาลัยราชภัฎสวนสุนันทา

ข้อความที่ปรากฏในบทความแต่ละเรื่องในวารสารวิชาการเล่มนี้เป็นความคิดเห็นส่วนตัวของผู้เขียนแต่ละท่านไม่เกี่ยวข้องกับมหาวิทยาลัยราชภัฎสวนสุนันทา และคณาจารย์ท่านอื่นๆในมหาวิทยาลัยฯ แต่อย่างใด ความรับผิดชอบองค์ประกอบทั้งหมดของบทความแต่ละเรื่องเป็นของผู้เขียนแต่ละท่าน หากมีความผิดพลาดใดๆ ผู้เขียนแต่ละท่านจะรับผิดชอบบทความของตนเองแต่ผู้เดียว