Journal of Platform Technology

  • 제11권1호
  • /
  • Pages.38-51
  • /
  • 2023
  • /
  • 2289-0181(pISSN)
  • /
  • 2289-019X(eISSN)
ICT플랫폼학회 (ICT Platform Society)
권호 표지

인공지능 기반 손 체스처 인식 정보를 활용한 지능형 인터페이스

Intelligent interface using hand gestures recognition based on artificial intelligence

  • 조항준 (전주대학교 스마트미디어학과) ;
  • 유준우 (전주대학교 스마트미디어학과) ;
  • 김은수 (전주대학교 전기전자공학과) ;
  • 이영재 (전주대학교 스마트미디어학과)
  • 투고 : 2023.02.13
  • 심사 : 2023.02.28
  • 발행 : 2023.02.28
PDF 다운로드
⟨ 이전 논문 다음 논문 ⟩

초록

인공지능에 기반한 손 제스처 인식 정보를 활용한 지능형 인터페이스 알고리즘을 제안한다. 이 방법은 기능적으로 사용자 손 제스처의 추적 및 인식을 미디어파이프와 KNN, LSTM, CNN의 인공지능 기법을 사용해 다양한 동작을 빠르고 지능적으로 인식되는 인터페이스이다. 제안한 알고리즘 성능 평가를 위해 자체 제작한 2D 탑뷰 레이싱 게임과 로봇제어에 적용한다. 알고리즘 적용 결과 게임의 가상 객체의 다양한 움직임을 세밀하고 강건하게 제어할 수 있었으며, 실세계의 로봇 제어에 적용한 결과 이동과 정지, 좌회전, 우회전 등의 제어가 가능하였다. 또한 게임의 메인 캐릭터와 실세계 로봇을 동시에 제어하여 가상과 현실의 공존공간 상황 제어를 위한 지능형 인터페이스로 최적화된 동작도 구현하였다. 제안한 알고리즘은 신체를 활용한 자연스럽고 직관적 특성과 손가락의 미세한 움직임 인식에 따른 정교한 제어가 가능하며, 빠른 기간 내에 숙련되는 장점이 있어 지능형 사용자 인터페이스 개발을 위한 기본자료로 활용될 수 있다.

We propose an intelligent interface algorithm using hand gesture recognition information based on artificial intelligence. This method is functionally an interface that recognizes various motions quickly and intelligently by using MediaPipe and artificial intelligence techniques such as KNN, LSTM, and CNN to track and recognize user hand gestures. To evaluate the performance of the proposed algorithm, it is applied to a self-made 2D top-view racing game and robot control. As a result of applying the algorithm, it was possible to control various movements of the virtual object in the game in detail and robustly. And the result of applying the algorithm to the robot control in the real world, it was possible to control movement, stop, left turn, and right turn. In addition, by controlling the main character of the game and the robot in the real world at the same time, the optimized motion was implemented as an intelligent interface for controlling the coexistence space of virtual and real world. The proposed algorithm enables sophisticated control according to natural and intuitive characteristics using the body and fine movement recognition of fingers, and has the advantage of being skilled in a short period of time, so it can be used as basic data for developing intelligent user interfaces.

키워드

참고문헌

  1. Song Bok-Deuk, Lee Seung-Hwan, Choi Hong-Kyu and Kim Sung-Hun, "Design and Implementation of a Stereoscopic Image Control System based on User Hand Gesture Recognition", Korea Institute of information and Communication Engineering, v.26 no.3, pp. 396-402, 2022.
  2. Heo Gyeong-Yong, Song Bok-Deuk, Kim Ji-Hong, "Hierarchical Hand Pose Model for Hand Expression Recognition", Korea Institute of information and Communication Engineering, v.25 no.10, pp. 1323-1329, 2021.
  3. Lim Young-Jae, Jung Il-Hong, "Hand Gesture Recognition in the Virtual Space based on Deep Learning", Korea Digital Contents Society, v.21 no.3, pp. 471-478, 2020. https://doi.org/10.9728/dcs.2020.21.3.471
  4. Kim Kyu-Min, Baek Joong-Hwan, "Real-time hand gesture recognition based on deep learning", Korea Multimedia Society, v.22 no.4, pp. 424-431, 2019.
  5. Lee Byeong-Hee, Oh Dong-Han, Kim Tae-Young, "3D Virtual Reality Game with Deep Learningbased Hand Gesture Recognition", Korea Computer Graphics Society, v.24 no.5, pp. 41-48, 2018. https://doi.org/10.15701/kcgs.2018.24.5.41
  6. P. K. Pisharady and M. Saerbeck, "Recent methods and databases in vision-based hand gesture recognition: a review," Computer Vision and Image Understanding, vol. 141, no. 4356, pp. 152-165, 2015. https://doi.org/10.1016/j.cviu.2015.08.004
  7. Kim Jin-young, Shim Hyun, "Development of Sign Language Learning Assistance System Using MediaPipe for Sign Language Education for the Hearing Impaired", Korea Electronics and Telecommunications Society, v.16 no.6 , pp.1355-1362 , 2021.
  8. Y. Zhou, M. Habermann, W. Xu, I. Habibie, C. Theobalt, F.Xu ,"Monocular Real-Time Hand Shape and Motion Capture Using Multi-Modal Data," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 5345-5354.
  9. Bread type developing countries website, https://www.youtube.com/@bbanghyong
  10. B. D. Song, H. K. Choi, S. H. Kim, "Research Trends of User Hand Gesutre Recognition Technologies for Utilizing User Interaction in Stereoscopic Images," in Proceeding of the Korea Institute of Communications and Information Sciences Summer Conference, 2021, pp. 119-120.
  11. MediaPipe, https://mediapipe.dev/
  12. E. Stergiopoulou, N. Papamarkos,"Hand gesture recognition using a neural network shape fitting technique", Engineering Applications of Artificial Intelligence, vol.22, no8, pp 1141-1158, 2009 https://doi.org/10.1016/j.engappai.2009.03.008
  13. A. Kendon, "Current issues in the study of gesture, in: The Biological Foundation of Gestures: Motor and Semiotic Aspects", Psychology Press, 1986, pp. 23-47.
  14. Different approaches to Computer Visual-Based Hand Gesture Recognition, https://medium.com/@bluebirb/different-approaches-to-computer-visual-based-hand-gesturerecognition-17164988e7f6
  15. Ha-Dang Ho, Hong-Quan Nguyen, Thuy-Binh Nguyen, Sinh-Thuong Vu, Thi-Lan Le, "Dynamic Hand Gesture Recognition from Egocentric Videos based on SlowFast Architecture", 2022 AsiaPacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), 2022, pp.01-07.
  16. Yongfeng Dong, Jielong Liu, Wenjie Yan, "Dynamic Hand Gesture Recognition Based on Signals From Specialized Data Glove and Deep Learning Algorithms", IEEE Transactions on Instrumentation and Measurement, vol.70, pp.1-14, 2021. https://doi.org/10.1109/TIM.2021.3077967
  17. MediaPipe Framework for ML solutions, https://viso.ai/computer-vision/mediapipe/
  18. MediaPipe Hands, https://google.github.io/mediapipe/solutions/hands.html
  19. MediaPipe Face Detection, https://google.github.io/mediapipe/solutions/face_detection.html
  20. Concept of K-Nearest Neighbor, https://hleecaster.com/ml-knn-concept
  21. Introduction to LSTM: https://www.analyticsvidhya.com/blog/2021/03/introduction-to-long-shortterm-memory-lstm/
  22. CNN concept : https://www.ibm.com/topics/convolutional-neural-networks