Grasp Recognition Development Using 16 Degree of Freedom Sensors and Convolutional Neural Network
Keywords:
Glove Sensor, Hand Grasp Recognition, Flex Sensor, Convolution Neural NetworkAbstract
It is recently challenging to use various sensors to interface data in Human-Computer Interaction (HCI) for developing hand gesture recognition systems. However, the previous sensor system has not met the hand gesture recognition accuracy for the machine vision. Therefore, this article presents the development of hand gesture recognition with a 16 Degree of Freedom (Dof) glove sensor combined with a convolution neural network. The flex sensors were installed to 16 points of the pivot point of the human hand on the glove so that each knuckle flex could be measured while holding the object. The 16 independent sensor multiplexer circuits and the adjustable buffer circuit were developed for use in this research to work with the Arduino Nano microcontroller to acquire each sensor's signal data to the computer. These signals are then converted to image data using Nearest interpolation and Color Mapping. The convolutional neural network techniques are used to recognize the hand gesture recognition systems. There were 4,000 hands-on captures of 20 hand grasps that were used to test the proposed system. The result indicated that the proposed system works very well, and recognition efficiency is 99.70%.
References
M. A. Ahmed, B. B. Zaidan, A. A. Zaidan, M. M. Salih, and M. M. Lakulu, “A Review on Systems-Based Sensory Gloves for Sign Language Recognition State of the Art between 2007 and 2017,” Sensors, Vol. 18, no. 7, pp. 2208, Jul. 2018.
B. Hu, and J. Wang, “Deep Learning Based Hand Gesture Recognition and UAV Flight Controls,” International Journal of Automation and Computing, Vol. 17, no.1, pp.17-29, Feb. 2020.
S. Sungtaea, T. Rezab, and L. Rezaa, “EMG and IMU based real‐time HCI using dynamic hand gestures for a multiple‐DoF robot arm,” Journal of Intelligent & Fuzzy Systems, Vol. 35, no. 1, pp. 861‐876, Jul. 2018.
M. Kim, S. H. Choi, K. B. Park, and J. Y. Lee, “User Interactions for Augmented Reality Smart Glasses: A Comparative Evaluation of Visual Contexts and Interaction Gestures,” Applied Sciences, Vol. 9, no.15, pp. 3171, Aug. 2019.
S. Makris, P. Tsarouchi, A. S. Matthaiakis, A. Athanasatos, X. Chatzigeorgiou, M. Stefos, K. Giavridis, and S. Aivaliotis, “Dual arm robot in cooperation with humans for flexible assembly,” CIRP Annals - Manufacturing Technology, Vol. 66, no. 1, pp. 13-16, May. 2017.
S. Lee, B. Jamil, S. Kim, and Y. Choi, “Fabric Vest Socket with Embroidered Electrodes for Control of Myoelectric Prosthesis,” Sensors, Vol. 20, no. 4, pp. 1196, Feb. 2020.
S. Lee, M. O. Kim, T. Kang, J. Park, and Y. Choi “Knit Band Sensor for Myoelectric Control of Surface EMG-Based Prosthetic Hand,” IEEE Sensors Journal, Vol. 18, no. 20, pp. 8578-8586, Oct. 2018.
I. D. Terrer, F. T. Alonso, A. l. R. Guzmán, M. A. Aznar, C. Alcubilla, S. P. Nombela, A. A. Espinosa, B. P. López, and A. G. Agudo, “Upper limb rehabilitation after spinalcord injury: a treatment based on a dataglove and an immersive virtual reality environment,” Disability and Rehabilitation: Assistive Technology, Vol. 11, no. 6, pp. 462-467, Jul. 2016.
F. Bin, S. Fuchun, L. Huaping, and G. Di, “A novel data glove using inertial and magnetic sensors for motion capture and robotic arm-hand teleoperation,” Industrial Robot: An International Journal, Vol. 44, no. 2, pp. 155-165, Mar. 2017.
M. Kazi and M. Bill, Robotic Hand Controlled by Glove Using Wireless Communication, KTH Royal Institute of Technology, SWEDEN.
S. S. Rautaray and A. Agrawal, “Vision based hand gesture recognition for human computer interaction: a survey,” Artificial Intelligence Review, Vol. 43, pp. 1-54, Jan. 2015.
F. Weichert, D. Bachmann, B. Rudak, and D. Fisseler, “Analysis of the Accuracy and Robustness of the Leap Motion Controller,” Sensors, Vol. 13, no.5, pp. 6380-6393, May. 2013.
A. J. Yánez, M. E. Benalcázar, and E. M. Maldonado, “Real-Time Hand Gesture Recognition Using Surface Electromyography and Machine Learning: A Systematic Literature Review,” Sensors, Vol. 20, no. 9, pp. 2467, Apr. 2020.
Y. Zhang, Y. Chen, H. Yu, X. Yang, and W. Lu, “Learning Effective Spatial–Temporal Features for sEMG Armband-Based Gesture Recognition,” IEEE Internet of Things Journal, Vol. 7, no. 8, pp. 6979-6992, Aug. 2020.
L. Dipietro, A. M. Sabatini, and P. Dario, “A Survey of Glove-Based Systems and Their Applications,” IEEE Transactions on Systems, Man, and Cybernetics: Systems, Vol. 38, no. 4, pp. 461-482, Jul. 2008.
J. Henderson, J. Condell, J. Connolly, D. Kelly, and K. Curran, “Review of Wearable Sensor-Based Health Monitoring Glove Devices for Rheumatoid Arthritis,” Sensors, Vol. 21, no. 5, pp. 1576, Feb. 2021.
D. Sim, Y. Baek, M. Cho, S. Park, A. S. M. S. Sagar, and H. S. Kim, “Low-Latency Haptic Open Glove for Immersive Virtual Reality Interaction,” Sensors, Vol. 21, no. 11, pp. 3682, May. 2021.
Y. Park, J. Lee, and J. Bae, “Development of a Finger Motion Measurement System using Linear Potentiometers,” in Proc. IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM), pp. 125-130, Jul. 2014.
J. Bang, J. You, and Y. Lee, “A Prototype of Flex Sensor Based Data Gloves to Track the Movements of Fingers,” Smart Media Journal, Vol. 8, no. 4, pp. 53-57, Nov. 2019.
Y. Zheng, Y. Peng, G. Wang, X. Liu, X. Dong, and J. Wanga, “Development and evaluation of a sensor glove for hand function assessment and preliminary attempts at assessing hand coordination,” Measurement, Vol. 93, pp.1-12, Jun. 2016.
S. Ciotti, E. Battaglia, N. Carbonaro, A. Bicchi, A. Tognetti, and M. Bianchi, “A Synergy-Based Optimally Designed Sensing Glove for Functional Grasp Recognition,” Sensors, Vol. 16, no. 6, pp. 811, Jun. 2016.
G.Saggio, “A novel array of flex sensors for a goniometric glove,” Sensors and Actuators A: Physical, Vol. 205, pp. 119-125, Nov. 2014.
S. A. A. S. M. Ali, N. S. Ahmad, and P. Goh “Flex Sensor Compensator via Hammerstein–Wiener Modeling Approach for Improved Dynamic Goniometry and Constrained Control of a Bionic Hand,” Sensors, Vol. 19, no. 18, pp. 3896, Sep. 2019.
S. Shin, H. U. Yoon, and B. Yoo “Hand Gesture Recognition Using EGaIn-Silicone Soft Sensors,” Sensors, Vol. 21, no. 9, pp. 3204, May. 2021.
S. Lee, Y. Choi, M. Sung, J. Bae, and Y. Choi “A Knitted Sensing Glove for Human Hand Postures Pattern Recognition,” Sensors, Vol. 21, no. 4, pp. 1364, Feb. 2021.
E. Fujiwara, D. Y. Miyatake, M. F. M. Santos, and C. K. Suzuki “Development of a Glove-Based Optical Fiber Sensor for Applications in Human-Robot Interaction,” in Proc. ACM/IEEE International Conference on Human-Robot Interaction (HRI), pp. 123-124, Mar. 2013.
Y. Li, H. Di, Y. Xin, and X. Jiang, “Optical fiber data glove for hand posture capture,” Optik, Vol. 233, pp. 166603, Feb. 2021.
L. Dipietro, A. M. Sabatini, and P. Dario, “Evaluation of an instrumented glove for hand-movement acquisition,” Journal of Rehabilitation Research and Development, Vol. 40, no. 2, pp. 179-190, Apr. 2003.
C. C. Yang and Y. L. Hsu, “A Review of Accelerometry-Based Wearable Motion Detectors for Physical Activity Monitoring,” Sensors, Vol. 10, no. 8, pp. 7772-7788, Aug. 2010.
B. S. Lin, I. J. Lee, S. Y. Yang, Y. C. Lo, J. Lee, and J. L. Chen, “Design of an Inertial-Sensor-Based Data Glove for Hand Function Evaluation,” Sensors, Vol. 18, no. 5, pp. 1545, May. 2018.
CyberGlove Systems LLC, “MoCap Glove System,” 2010. [Online]. Available: https://static1.squarespace.com/static/559c381ee4b0ff7423b6b6a4/t/5602fbc3e4b07ebf58d47e34/1443036099686/CyberGlove+III+DataSheet.pdf. [Accessed Oct. 1, 2021].
5DT Inc.,“5DT Data Glove Ultra Series,” Oct. 2004. [Online]. Available: https://www.5dt.com/downloads/dataglove/ultra/5DT Data Glove Ultra Manual v1.3.pdf. [Accessed Oct. 1, 2021].
T. L. Baldi, S. Scheggi, L. Meli, M. Mohammadi, and D. Prattichizzo, “GESTO: a Glove for Enhanced Sensing and Touching Based on Inertial and Magnetic Sensors for Hand Tracking and Cutaneous Feedback,” IEEE Transactions on Human-Machine Systems, Vol. 47, no. 6, pp. 1066-1076, Dec. 2017.
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet Classification with Deep Convolutional Neural Networks,” Communications of the ACM, Vol. 60, no. 6, pp. 84-90, Jun. 2017.
S. Bianco, R. Cadene, L. Celona, and P. Napoletano, “Benchmark Analysis of Representative Deep Neural Network Architectures,” IEEE Access, Vol. 6, pp. 64270-6427, Nov. 2018.
M. Cai, K. M. Kitani, and Y. Sato, “An Ego-Vision System for Hand Grasp Analysis,” IEEE Transactions on Human-Machine Systems, Vol. 47, no. 4, pp. 524-535, Aug. 2017.
T. Feix, J. Romero, H. B. Schmiedmayer, A. M. Dollar, and D. Kragic, “The GRASP Taxonomy of Human Grasp Types,” IEEE Transactions on Human-Machine Systems, Vol. 46, no. 1, pp. 66-77, Feb. 2016.
R. Bray, “Sensor Workshop at ITP: Reports / Flex,” 2012. [Online]. Available: https://itp.nyu.edu/archive/physcomp-spring2014/sensors/Reports/Flex.html. [Accessed Oct. 1, 2021].
Electronics Source Co.,Ltd, “Arduino Nano 3.1,” [Online]. Available: https://www.es.co.th/Schemetic/PDF/ARMB-0022.PDF. [Accessed Oct. 1, 2021].
Downloads
Published
Issue
Section
License
ลิขสิทธิ์ของบทความที่ตีพิมพ์ในวารสารฉบับนี้จะยังเป็นของผู้แต่งและยินยอมให้สิทธิ์เผยแพร่กับทางวารสาร
การเผยแพร่ในระบบวารสารแบบเปิดนี้ บทความจะสามารถนำไปใช้ได้ฟรีในการศึกษา และในทางที่ไม่เกี่ยวกับการค้า
