Science of Emotion and Sensibility (감성과학)

Korean Society for Emotion and Sensibility (한국감성과학회)
권호 표지
DOI QR코드

DOI QR Code

Classification of Negative Emotions based on Arousal Score and Physiological Signals using Neural Network

신경망을 이용한 다중 심리-생체 정보 기반의 부정 감성 분류

  • Kim, Ahyoung (Bio-Medical IT Convergence Research Division, Electronics and Telecommunications Research Institute) ;
  • Jang, Eun-Hye (Bio-Medical IT Convergence Research Division, Electronics and Telecommunications Research Institute) ;
  • Sohn, Jin-Hun (Department of Psychology, Brain Research Institute, Chungnam National University)
  • 김아영 (한국전자통신연구원 바이오의료IT연구본부) ;
  • 장은혜 (한국전자통신연구원 바이오의료IT연구본부) ;
  • 손진훈 (충남대학교 심리학과/뇌과학연구소)
  • Received : 2018.02.26
  • Accepted : 2018.03.26
  • Published : 2018.03.31
Download PDF
⟨ Previous Next ⟩

Abstract

The mechanism of emotion is complex and influenced by a variety of factors, so that it is crucial to analyze emotion in broad and diversified perspectives. In this study, we classified neutral and negative emotions(sadness, fear, surprise) using arousal evaluation, which is one of the psychological evaluation scales, as well as physiological signals. We have not only revealed the difference between physiological signals coupled to the emotions, but also assessed how accurate these emotions can be classified by our emotional recognizer based on neural network algorithm. A total of 146 participants(mean age $20.1{\pm}4.0$, male 41%) were emotionally stimulated while their physiological signals of the electrocardiogram, blood flow, and dermal activity were recorded. In addition, the participants evaluated their psychological states on the emotional rating scale in response to the emotional stimuli. Heart rate(HR), standard deviation(SDNN), blood flow(BVP), pulse wave transmission time(PTT), skin conduction level(SCL) and skin conduction response(SCR) were calculated before and after the emotional stimulation. As a result, the difference between physiological responses was verified corresponding to the emotions, and the highest emotion classification performance of 86.9% was obtained using the combined analysis of arousal and physiological features. This study suggests that negative emotion can be categorized by psychological and physiological evaluation along with the application of machine learning algorithm, which can contribute to the science and technology of detecting human emotion.

감성은 복잡하고 다양한 요인들에 의해 영향을 받기 때문에 다각적인 측면에서 고려되어야 한다. 본 연구에서는 심리 평가 척도의 하나인 각성(arousal) 지표와 다중 생체신호에서 추출된 생체지표 반응을 이용하여 중립 및 부정 감성(슬픔, 공포, 놀람)의 분류하였다. 이를 위하여 감성에 따른 생체지표 반응의 차이를 확인하였고, 다중 신경망 알고리즘 기반의 감성 인식기를 적용하여 이들 감성이 얼마나 정확하게 분류되는가를 확인하였다. 총 146명의 실험 참가자(평균 연령 $20.1{\pm}4.0$, 남성 41%)를 대상으로 감성 유발 자극을 제시하고 동시에 생체신호(심전도, 혈류맥파, 피부전기활동)를 측정하였다. 또한 감성 유발 자극에 대한 심리 반응을 감성 평가 척도로 평가하였다. 측정된 생체신호에서 심박률(HR), NN 간격의 표준편차(SDNN), 혈류량(BVP), 맥파전달시간(PTT), 피부전도수준(SCL), 피부전도반응(SCR)을 추출하였다. 결과 분석을 위하여 감성 자극에 대한 각성도와 안정 상태와 감성 상태의 생체지표 반응을 활용하였다. 또한 감성 분류를 위하여 다중 신경망 기반의 감성 인식기를 활용하였다. 그 결과, 감성에 따른 생체지표 반응의 차이를 확인하였고, 이들 감성의 분류 성능은 각성도와 모든 생체지표 특징들을 조합하였을 때 정확도가 가장 높음(86.9%)을 확인하였다. 본 연구는 심리 및 생체지표 추출과 기계학습 기술의 적용을 통하여 부정 감성을 분류할 수 있음을 제안하며, 이는 인간의 감성을 탐지하는 감성 인식 기술을 확립하는데 기여할 것으로 예상한다.

Keywords

References

  1. Badcock, N. A., Preece, K. A., de Wit, B., Glenn, K., Fieder, N., Thie, J., & McArthur, G. (2015). Validation of the Emotiv EPOC EEG system for research quality auditory event-related potentials in children. PeerJ, 3, e907. DOI: 10.7717/ peerj.907
  2. Banziger, T., Grandjean, D., & Scherer, K. R. (2009). Emotion recognition from expressions in face, voice, and body: the Multimodal Emotion Recognition Test (MERT). Emotion (Washington, D.C.), 9(5), 691-704. DOI: 10.1037/a0017088
  3. Basu, S., Jana, N., Bag, A. M. M., Mukherjee, J., Kumar, S., & Guha, R. (2015). Emotion recognition based on physiological signals using valence-arousal model. In 2015 Third International Conference on Image Information Processing(ICIIP)(pp. 50-55). DOI: 10.1109/ICIIP.2015.7414739
  4. Calvo, R. A., & D'Mello, S. (2010). Affect detection: An interdisciplinary review of models, methods, and their applications. IEEE Transactions on Affective Computing, 1(1), 18-37. DOI: 10.1109/T-AFFC.2010.1
  5. Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., & Taylor, J. G. (2001). Emotion recognition in human-computer interaction. IEEE Signal Processing Magazine, 18(1), 32-80. DOI: 10.1109/79.911197
  6. Duan, R. N., Zhu, J. Y., & Lu, B. L. (2013). Differential entropy feature for EEG-based emotion classification. In 2013 6th International IEEE/EMBS Conference on Neural Engineering(NER) (pp. 81-84). DOI: 10.1109/NER.2013.6695876
  7. Ebner, N. C., & Fischer, H. (2014). Emotion and aging: evidence from brain and behavior. Frontiers in Psychology, 5. DOI: 10.3389/fpsyg.2014.00996
  8. Ekman, P. (1992). An argument for basic emotions. Cognition & Emotion, 6, 169-200. DOI: 10.1080/02699939208411068
  9. Jang, E.-H., Park, B.-J., Park, M.-S., Kim, S.-H., & Sohn, J.-H. (2015). Analysis of physiological signals for recognition of boredom, pain, and surprise emotions. Journal of Physiological Anthropology, 34(1), 25. DOI: 10.1186/s40101-015-0063-5
  10. Jirayucharoensak, S., Pan-Ngum, S., & Israsena, P. (2014). EEG-based emotion recognition using deep learning network with principal component based covariate shift adaptation. The Scientific World Journal, 627892. DOI: 10.1155/2014/627892
  11. Kreibig, S. D. (2010). Autonomic nervous system activity in emotion: a review. Biological Psychology, 84(3), 394-421. DOI: 10.1016/j.biopsycho.2010.03.010
  12. Levenson, R. W., Ekman, P., & Friesen, W. V. (1990). Voluntary facial action generates emotion-specific autonomic nervous system activity. Psychophysiology, 27(4), 363-384. Retrieved from https://www.ncbi.nlm.nih.gov/pubmed/2236440 https://doi.org/10.1111/j.1469-8986.1990.tb02330.x
  13. Lin, Y. P., Wang, C. H., Wu, T. L., Jeng, S. K., & Chen, J. H. (2009). EEG-based emotion recognition in music listening: A comparison of schemes for multiclass support vector machine. In 2009 IEEE International Conference on Acoustics, Speech and Signal Processing (pp. 489-492). DOI: 10.1109/ICASSP.2009.4959627
  14. Lindquist, K. A., MacCormack, J. K., & Shablack, H. (2015). The role of language in emotion: Predictions from psychological constructionism. Frontiers in Psychology, 6, 444. DOI: 10.3389/fpsyg.2015.00444
  15. Majumder, S., Mondal, T., & Deen, M. J. (2017). Wearable sensors for remote health monitoring. Sensors (Basel, Switzerland), 17(1). DOI: 10.3390/s17010130
  16. Mill, A., Allik, J., Realo, A., & Valk, R. (2009). Agerelated differences in emotion recognition ability: a cross-sectional study. Emotion (Washington, D.C.), 9(5), 619-630. DOI: 10.1037/a0016562
  17. Nasoz, F., Alvarez, K., Lisetti, C. L., & Finkelstein, N. (2004). Emotion recognition from physiological signals using wireless sensors for presence technologies. Cognition, Technology & Work, 6(1), 4-14. DOI: 10.1007/s10111-003-0143-x
  18. Picard, R. W. (2003). Affective computing: challenges. International Journal of Human-Computer Studies, 59(1), 55-64. DOI: 10.1016/S1071-5819(03)00052-1
  19. Picard, R. W., Vyzas, E., & Healey, J. (2001). Toward machine emotional intelligence: analysis of affective physiological state. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(10), 1175-1191. DOI: 10.1109/34.954607
  20. Russell, J. A. (1980). A circumplex model of affect. Journal of Personality and Social Psychology, 39(6), 1161-1178. DOI: 10.1037/h0077714
  21. Simon, E. W., Rosen, M., Grossman, E., & Pratowski, E. (1995). The relationships among facial emotion recognition, social skills, and quality of life. Research in Developmental Disabilities, 16(5), 383-391. Retrieved from https://www.ncbi.nlm.nih.gov/pubmed/8532917 https://doi.org/10.1016/0891-4222(95)00025-I
  22. Tan, J.-W., Andrade, A. O., Li, H., Walter, S., Hrabal, D., Rukavina, S., Limbrecht-Ecklundt, K., Hoffman, H., Traue, H. C. (2016). Recognition of intensive valence and arousal affective states via facial electromyographic activity in young and senior adults. PLoS ONE, 11(1), e0146691. DOI: 10.1371/journal.pone.0146691
  23. Wagner, J., Kim, J., & Andre, E. (2005). From physiological signals to emotions: Implementing and comparing selected methods for feature extraction and classification. In 2005 IEEE International Conference on Multimedia and Expo(pp. 940-943). DOI: 10.1109/ICME.2005.1521579
  24. Wiem, M. B. H., & Lachiri, Z. (2016). Emotion assessing using valence-arousal evaluation based on peripheral physiological signals and support vector machine. In 2016 4th International Conference on Control Engineering Information Technology(CEIT) (pp. 1-5). DOI: 10.1109/CEIT.2016.7929117
  25. Zaja, R. H., & Rojahn, J. (2008). Facial emotion recognition in intellectual disabilities. Current Opinion in Psychiatry, 21(5), 441-444. DOI: 10.1097/YCO.0b013e328305e5fd
  26. Zhang, Q., Chen, X., Zhan, Q., Yang, T., & Xia, S. (2017). Respiration-based emotion recognition with deep learning. Computers in Industry, 92-93, 84-90. DOI: 10.1016/j.compind.2017.04.005
  27. Park, M. S., Kim, H. E., & Sohn, J. H. (2011). Development of emotion-evoking stimuli to provoke spontaneous emotions. Proceedings for the 2011 Annual Spring Conference of Korean Society for Emotion & Sensibility (pp. 505-512). Daejeon, Republic of Korea. Retrieved from http://www.koses.or.kr/
  28. Lee, K. H. (1997). Human sensibility and its measurement and evaluation. Annual Conference Papers of Korean Society for Emotion & Sensibility (pp. 37-42). Daejeon, Republic of Korea. Retrieved from http://www.koses.or.kr/