Surface Electromyography-based Gesture Recognition for Robot Hand Control
Abstract
This paper presents a human-machine interaction interface for a teleoperated control of a three-finger robot hand based on a surface electromyography. Based on the recorded sEMG signals, motion intention of a human operator is recognized and mapped onto corresponding predefined robot hand grasps. First, the experimental setup and underlying methodology of dataset generation are presented. Namely, 4-channel EMG data were collected from the forearm muscles of 4 healthy subjects as they performed a sequence of predefined grasps. Data processing included: data filtering and segmentation, feature extraction both in time and frequency domains, data annotation, and gestures classification. Two types of classifiers (with multiple variations of parameters) were implemented and compared: (i) Support Vector Machines with multiple different kernels and (ii) K-Nearest-Neighbors. The k-cross validation procedure was combined with the holdout method to generate training, validation and test sets. Performance of each classifier is presented in the form of a confusion matrix and a set of statistical measures, including F-measure. Finally, the integration of the proposed sEMG-based HMI interface and the selected classifier with a robot hand motion controller is presented.
References
[2] Lii Neal Y., Pereira Aaron, Dietl Julian, Stillfried Georg, Schmidt Annika, Beik-Mohammadi Hadi, Baker Thomas, Maier Annika, Pleintinger Benedikt, Chen Zhaopeng, Elawad Amal, Mentzer Lauren, Pineault Austin, Reisich Philipp, Albu-Schäffer Alin,” Exodex Adam—A Reconfigurable Dexterous Haptic User Interface for the Whole Hand”, Front. Robot. AI, Vol 8, March 2022.
[3] S. Li et al., "Vision-based Teleoperation of Shadow Dexterous Hand using End-to-End Deep Neural Network," 2019 International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada, 2019, pp. 416-422, doi: 10.1109/ICRA.2019.8794277.
[4] A. Handa et al., "DexPilot: Vision-Based Teleoperation of Dexterous Robotic Hand-Arm System," 2020 IEEE International Conference on Robotics and Automation (ICRA), Paris, France, 2020, pp. 9164-9170, doi: 10.1109/ICRA40945.2020.9197124.
[5] B. Fang, D. Guo, F. Sun, H. Liu and Y. Wu, "A robotic hand-arm teleoperation system using human arm/hand with a novel data glove," 2015 IEEE International Conference on Robotics and Biomimetics (ROBIO), Zhuhai, China, 2015, pp. 2483-2488, doi: 10.1109/ROBIO.2015.7419712.
[6] Caeiro-Rodríguez M, Otero-González I, Mikic-Fonte FA, Llamas-Nistal M. A Systematic Review of Commercial Smart Gloves: Current Status and Applications. Sensors (Basel). 2021 Apr 10;21(8):2667. doi: 10.3390/s21082667. PMID: 33920101; PMCID: PMC8070066.
[7] L. Dipietro, A. M. Sabatini and P. Dario, "A Survey of Glove-Based Systems and Their Applications," in IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), vol. 38, no. 4, pp. 461-482, July 2008, doi: 10.1109/TSMCC.2008.923862.
[8] J. Vogel, C. Castellini and P. van der Smagt, "EMG-based teleoperation and manipulation with the DLR LWR-III," 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems, San Francisco, CA, USA, 2011, pp. 672-678, doi: 10.1109/IROS.2011.6094739.
[9] R. Meattini, S. Benatti, U. Scarcia, D. De Gregorio, L. Benini and C. Melchiorri, "An sEMG-Based Human–Robot Interface for Robotic Hands Using Machine Learning and Synergies," in IEEE Transactions on Components, Packaging and Manufacturing Technology, vol. 8, no. 7, pp. 1149-1158, July 2018, doi: 10.1109/TCPMT.2018.2799987.
[10] J. Lee, K. Kwon and W.H. Yeo, “Recent advances in wearable exoskeletons for human strength augmentation”, Flex. Print. Electron. 7, 1-31, (2022), DOI 10.1088/2058-8585/ac6a96
[11] Al-Shamayleh, A.S., Ahmad, R., Abushariah, M.A.M. et al. A systematic literature review on vision based gesture recognition techniques. Multimed Tools Appl 77, 28121–28184 (2018). https://doi.org/10.1007/s11042-018-5971-z
[12] L. Guo, Z. Lu and L. Yao, "Human-Machine Interaction Sensing Technology Based on Hand Gesture Recognition: A Review," in IEEE Transactions on Human-Machine Systems, vol. 51, no. 4, pp. 300-309, Aug. 2021, doi: 10.1109/THMS.2021.3086003.
[13] H.K. Hameed, W.Z. Hassan, S. Shafie, S.A. Ahmad, and H. Jaafar, “A Review on Surface Electromyography-Controlled Hand Robotic Devices Used for Rehabilitation and Assistance in Activities of Daily Living”, JPO Journal of Prosthetics and Orthotic, 32(1), 3-13, (2020), DOI: 10.1097/JPO.0000000000000277
[14] R. Meattini, R. Suárez, G. Palli and C. Melchiorri, "Human to Robot Hand Motion Mapping Methods: Review and Classification," in IEEE Transactions on Robotics, vol. 39, no. 2, pp. 842-861, April 2023, doi: 10.1109/TRO.2022.3205510.
[15] R. Li, H. Wang and Z. Liu, "Survey on Mapping Human Hand Motion to Robotic Hands for Teleoperation," in IEEE Transactions on Circuits and Systems for Video Technology, vol. 32, no. 5, pp. 2647-2665, May 2022, doi: 10.1109/TCSVT.2021.3057992.
[16] Bitzer, Sebastian & van der Smagt, Patrick. (2006). Learning EMG control of a robotic hand: Towards Active Prostheses. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA 2006). 2006. 2819 - 2823. 10.1109/ROBOT.2006.1642128.
[17] Wan-Ting Shi, Zong-Jhe Lyu, Shih-Tsang Tang, Tsorng-Lin Chia, Chia-Yen Yang, “A bionic hand controlled by hand gesture recognition based on surface EMG signals: A preliminary study”, Biocybernetics and Biomedical Engineering, Volume 38, Issue 1, 2018, pp. 126-135,https://doi.org/10.1016/j.bbe.2017.11.001.
[18] Zheng N, Li Y, Zhang W and Du M (2022) User-Independent EMG Gesture Recognition Method Based on Adaptive Learning. Front. Neurosci. 16:847180. doi: 10.3389/fnins.2022.847180
[19] Gopal P, Gesta A, Mohebbi A. A Systematic Study on Electromyography-Based Hand Gesture Recognition for Assistive Robots Using Deep Learning and Machine Learning Models. Sensors. 2022; 22(10):3650. https://doi.org/10.3390/s22103650
[20] Parajuli N, Sreenivasan N, Bifulco P, Cesarelli M, Savino S, Niola V, Esposito D, Hamilton TJ, Naik GR, Gunawardana U, et al. “Real-Time EMG Based Pattern Recognition Control for Hand Prostheses: A Review on Existing Methods, Challenges and Future Implementation.”, Sensors, Vol. 19, No. 20, 4596, 2019, DOI: https://doi.org/10.3390/ s19204596
[21] A. H. Al-Timemy, G. Bugmann, J. Escudero and N. Outram, "Classification of Finger Movements for the Dexterous Hand Prosthesis Control With Surface Electromyography," in IEEE Journal of Biomedical and Health Informatics, vol. 17, no. 3, pp. 608-618, May 2013, doi: 10.1109/JBHI.2013.2249590.
[22] Rossi, M., Benatti, S., Farella, E. & Benini, L, “Hybrid EMG classifier based on HMM and SVM for hand gesture recognition in prosthetics”. 2015 IEEE International Conference on Industrial Technology (ICIT), Seville, Spain, pp. 1700–1705, 17-19 March 2015.
[23] Huang, Y., Englehart, K. B., Hudgins, B. & Chan, A. D. C,” A Gaussian mixture model based classification scheme for myoelectric control of powered upper limb prostheses”, IEEE Trans. Biomed. Eng, Vol. 52, No. 11, pp. 1801–1811, November 2005.
[24] Kim, EunSu, JaeWook Shin, YongSung Kwon, and BumYong Park, "EMG-Based Dynamic Hand Gesture Recognition Using Edge AI for Human–Robot Interaction" Electronics, Vol. 12, No. 7, pp. 1541, March 2023, DOI: https://doi.org/10.3390/electronics12071541
[25] Yang Z, Jiang D, Sun Y, Tao B, Tong X, Jiang G, Xu M, Yun J, Liu Y, Chen B and Kong J, “Dynamic Gesture Recognition Using Surface EMG Signals Based on Multi-Stream Residual Network”, Front. Bioeng. Biotechnol, Vol. 9, Article 779353,pp. 1-13, October 2021, DOI: 10.3389/fbioe.2021.779353, 2021.
[26] Lee, H., Lee, S., Kim, J. et al, “Stretchable array electromyography sensor with graph neural network for static and dynamic gestures recognition system”, npj Flex Electron, Vol. 7, No. 20, pp. 1-13, April 2023, DOI: https://doi.org/10.1038/s41528-023-00246-3
[27] C. M. Bishop, Pattern Recognition and Machine Learning, 1st ed, Springer New York, NY, 2006
[28] V. Kecman, “Learning and Soft Computing - Support Vector Machines, Neural Networks, and Fuzzy Logic Models”, The MIT Press, 2001.
[29] DRAKE robotic simulator [Internet], [Reviewed 19 June 2023, Cited 19 June 2023], Available at: https://drake.mit.edu/
I (we), the author(s), hereby declare under full moral, financial and criminal liability that the manuscript submitted for publication to the Journal of Computer and Forensic Sciences
a) is the result of my (our) own original research and that I (we) hold the right to publish it;
b) does not infringe any copyright or other third-party proprietary rights;
c) complies with the Journal’s research and publishing ethics standards;
d) has not been published elsewhere, under this or any other title;
e) is not under consideration by another publication, under this or any other title.
I (we) also declare under full moral, financial and criminal liability:
f) that all conflicts of interest that may directly or potentially influence or impart bias on the work have been disclosed in the manuscript;
g) that if the article has been accepted for publishing I (we) will transfer all copyright ownership of the manuscript to the University of Criminal Investigation and Police Studies in Belgrade.
Signed by the Corresponding Author on behalf of the all other authors.