International Journal of Computer Science & Network Security

International Journal of Computer Science & Network Security (국제컴퓨터통신보호논문지학회)
권호 표지
DOI QR코드

DOI QR Code

A Review of Facial Expression Recognition Issues, Challenges, and Future Research Direction

  • Yan, Bowen (School of Computer Science and Engineering, Taylor's University) ;
  • Azween, Abdullah (School of Computer Science and Engineering, Taylor's University) ;
  • Lorita, Angeline (School of Computer Science and Engineering, Taylor's University) ;
  • S.H., Kok (School of Computer Science and Engineering, Taylor's University)
  • Received : 2023.01.05
  • Published : 2023.01.30
Download PDF
⟨ Previous Next ⟩

Abstract

Facial expression recognition, a topical problem in the field of computer vision and pattern recognition, is a direct means of recognizing human emotions and behaviors. This paper first summarizes the datasets commonly used for expression recognition and their associated characteristics and presents traditional machine learning algorithms and their benefits and drawbacks from three key techniques of face expression; image pre-processing, feature extraction, and expression classification. Deep learning-oriented expression recognition methods and various algorithmic framework performances are also analyzed and compared. Finally, the current barriers to facial expression recognition and potential developments are highlighted.

Keywords

References

  1. Mehrabian. A, "Nonverbal communication," Routledge, 2017, pp.193-200.
  2. Ekman. P and Heider. K.G, "The universality of a contempt expression: A replication," Motiv Emot 12, 1988, pp.303-308. https://doi.org/10.1007/BF00993116
  3. S. Li and W. Deng, "Deep Facial Expression Recognition: A Survey," in IEEE Transactions on Affective Computing, vol.13, no.3, 2022, pp.1195-1215. https://doi.org/10.1109/TAFFC.2020.2981446
  4. M. Lyons, S. Akamatsu, M. Kamachi and J. Gyoba, "Coding facial expressions with Gabor wavelets," Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition, 1998, pp. 200-205.
  5. T. Kanade, J. F. Cohn and Yingli Tian, "Comprehensive database for facial expression analysis," Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580), 2000, pp. 46- 53, doi: 10.1109/AFGR.2000.840611.
  6. P. Lucey, J. F. Cohn, T. Kanade, J. Saragih, Z. Ambadar and I. Matthews, "The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression," 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops, 2010, pp. 94-101, doi: 10.1109/CVPRW.2010.5543262.
  7. M. Lyons, S. Akamatsu, M. Kamachi and J. Gyoba, "Coding facial expressions with Gabor wavelets," Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition, 1998, pp. 200-205, doi: 10.1109/AFGR.1998.670949.
  8. Lijun Yin, Xiaozhou Wei, Yi Sun, Jun Wang and M. J. Rosato, "A 3D facial expression database for facial behavior research," 7th International Conference on Automatic Face and Gesture Recognition (FGR06), 2006, pp. 211-216, doi: 10.1109/FGR.2006.6.
  9. Moreno. A and Sanchez. A, "Gavabdb: A 3d face database," In 2nd COST275 workshop on Biometrics on the Internet, 2004, pp. 75-80.
  10. X. Li, T. Pfister, X. Huang, G. Zhao and M. Pietikainen, "A Spontaneous Microexpression Database: Inducement, collection and baseline," 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), 2013, pp.1-6, doi: 10.1109/FG.2013.6553717.
  11. Yan W-J, Li X, Wang S-J, Zhao G, Liu Y-J, Chen Y-H, et al., "CASME II: An Improved Spontaneous Micro-Expression Database and the Baseline Evaluation," PLoS, vol.9, no.1, 2014.
  12. Goodfellow, I.J. et al., "Challenges in Representation Learning: A Report on Three Machine Learning Contests," In Neural Information Processing(ICONIP), vol.8228, 2013.
  13. Shan Li, Weihong Deng, and JunPing Du, "Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild," In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 2852-2861.
  14. Dhall, R. Goecke, S. Lucey, and T. Gedeon, "Static facial expressionanalysis in tough conditions: Data, evaluation protocol and benchmark," in Computer Vision Workshops (ICCV Workshops), 2011 IEEE Interna-tional Conference on. IEEE, 2011, pp. 2106-2112.
  15. Dhall, R. Goecke, S. Lucey, T. Gedeon et al., "Collecting large, richlyannotated facial-expression databases from movies," IEEE multimedia, vol. 19, no. 3, 2012, pp. 34-41. https://doi.org/10.1109/MMUL.2012.26
  16. Hallinan PW, Gordon G, Yuille AL, Giblin P, and Mumford D, "Two- and Three-Dimensional Patterns of the Face (1st ed.)," CRC Press, 1999.
  17. S. Phimoltares, C. Lursinsap and K. Chamnongthai, "Face detection and facial feature localization without considering the appearance of image context," Image and Vision Computing,Vol. 25, no. 5,2007,pp.741-753, doi:10.1016/j.imavis.2006.05.017.
  18. Wang, Jilin, Ye, Jian-Long, Zhao, Li, and Zou, Cai-Rong, "Face Detection Based on Curvilinear Wave Hidden Markov Model," Journal of Sensing Technology, vol. 24, no.5, 2011, pp.714-718.
  19. T.F. Cootes, C.J. Taylor, D.H. Cooper, and Graham. J, "Active Shape ModelsTheir Training and Application," Computer Vision and Image Understanding, Vol. 61, no.1, 1995, pp.38-59. https://doi.org/10.1006/cviu.1995.1004
  20. Xiong. X and De la Torre. F, "Supervised descent method and its applications to face alignment," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013, pp. 532-539
  21. Simard, Patrice Y., David Steinkraus, and John C. Platt, "Best practices for convolutional neural networks applied to visual document analysis," Icdar, vol.3, 2003.
  22. Krizhevsky. A, Sutskever. I, and Hinton. G. E, "Imagenet classification with deep convolutional neural networks," Communications of the ACM, vol.60, no.6, 2017, pp.84-90. https://doi.org/10.1145/3065386
  23. Andre Teixeira Lopes, Edilson de Aguiar, Alberto F. De Souza, and Thiago Oliveira-Santos, "Facial expression recognition with Convolutional Neural Networks: Coping with few data and the training sample order," Pattern Recognition, vol. 61, 2017, pp. 610-628. https://doi.org/10.1016/j.patcog.2016.07.026
  24. M.V. Zavarez, R.F. Berriel and T. Oliveira-Santos, "Cross-Database Facial Expression Recognition Based on Fine-Tuned Deep Convolutional Network," 2017 30th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), 2017, pp. 405-412.
  25. Yu. Z, and Zhang. C, "Image based static facial expression recognition with multiple deep network learning, " In Proceedings of the 2015 ACM on international conference on multimodal interaction,2015, pp. 435-442.
  26. B.-K. Kim, H. Lee, J. Roh, and S.-Y. Lee, "Hierarchical committeeof deep cnns with exponentially-weighted decision fusion for staticfacial expression recognition," in Proceedings of the 2015 ACM onInternational Conference on Multimodal Interaction. ACM, 2015, pp. 427-434.
  27. J. Li and E. Y. Lam, "Facial expression recognition using deep neural networks," 2015 IEEE International Conference on Imaging Systems and Techniques (IST), 2015, pp. 1-6, doi: 10.1109/IST.2015.7294547.
  28. Rui Huang, Shu Zhang, Tianyu Li, and Ran He, "Beyond face rotation: Global and local perception gan for photorealistic and identity preserving frontal view synthesis," in Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017, pp. 2439-2448
  29. M. Lyons, S. Akamatsu, M. Kamachi and J. Gyoba, "Coding facial expressions with Gabor wavelets," in Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition, 1998, pp. 200-205, doi: 10.1109/AFGR.1998.670949.
  30. Zhen Wen and Huang, "Capturing subtle facial motions in 3D face tracking," in Proceedings Ninth IEEE International Conference on Computer Vision, vol.2, 2003, pp.1343-1350, doi: 10.1109/ICCV.2003.1238646.
  31. Feng. X, Pietikainen. M, and Hadid. A, "Facial expression recognition based on local binary patterns," Pattern Recognit Image Anal, vol. 17, 2007, pp. 592- 598. https://doi.org/10.1134/S1054661807040190
  32. Yimo Guo, Guoying Zhao, and Matti Pietikainen, "Discriminative features for texture description," Pattern Recognition, vol.45, no.10, 2012, pp.3834-3843, doi:10.1016/j.patcog.2012.04.003.
  33. Satiyan. M, Hariharan. M, and Nagarajan. R, "Recognition of facial expression using Haar wavelet transform," Journal of Electrical and Electronics Systems Research (JEESR), Vol.3, no.11, 2010, pp. 89-96. https://doi.org/10.1109/ICIAS.2010.5716228
  34. M. Hu, K. Li, XH. Wang, and FJ. Ren, "Facial expression recognition based on histogram weighted HCBP," Journal of Electronic Measurement and Instrument, vol.29, no.7, 2015, pp. 953-960.
  35. M. Pantic and L. J. M. Rothkrantz, "Facial action recognition for facial expression analysis from static face images," in IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), vol. 34, no.3, 2004, pp. 1449-1461, doi: 10.1109/TSMCB.2004.825931.
  36. David Cristinacce, Tim Cootes, and Ian Scott, "A multi-stage ap-proach to facial featuredetection," Proceedings of the British Machine Vision Conference, 2004, pp.231-240.
  37. Y. Saatci and C. Town, "Cascaded classification of gender and facial expression using active appearance models," in 7th International Conference on Automatic Face and Gesture Recognition (FGR06), 2006, pp. 393-398, doi: 10.1109/FGR.2006.29.
  38. L. Ma and K. Khorasani, "Facial expression recognition using constructive feedforward neural networks," in IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), vol. 34, no. 3, pp. 1588-1595, June 2004, doi: 10.1109/TSMCB.2004.825930.
  39. Wu. XG, and Luo. LM, "An improved optical flow field calculation method," Acta electronica sinica, Vol. 1, 2000, pp.130-131.
  40. LI Dan, XIE Lun, LU Ting, HAN Jing, HU Bo, WANG Zhi-liang, and REN Fu-ji, "Capture of micro-expressions based on the entropy of oriented optical flow," Chinese Journal of Engineering, vol. 39, no. 11, 2017, pp.1727-1734.
  41. Huang YM, Zhang GB, Dong F, and Da FP, "Gabor-based, Fisher face multifeature extraction and integrated SVM for face expression recognition," Computer Application Research, vol. 28, no. 4, 2011, pp.1536-1539.
  42. Liu. P, Zhou. J.T, Tsang. I.WH, Meng. Z, Han. S, and Tong. Y, "Feature Disentangling Machine - A Novel Approach of Feature Selection and Disentangling in Facial Expression Analysis," Lecture Notes in Computer Science, vol. 8692, 2014, pp.151-166.
  43. Qi et al., "Facial Expressions Recognition Based on Cognition and Mapped Binary Patterns," in IEEE Access, vol. 6, pp. 18795-18803, 2018, doi: 10.1109/ACCESS.2018.2816044.
  44. Xu WH and Sun ZX, "An LSVM algorithm for video sequence expression classification," Journal of Computer-Aided Design and Graphics, vol. 21, no. 4, 2009, pp.542-548.
  45. WANG Xiaohua, XIA Chen, and Hu Min, "Video sequence facial expression recognition with spatio-temporal features," Journal of electronics and information, vol. 40, no.3, 2018, pp.626-632.
  46. K R. Kandemir and G. Ozmen, "FACIAL EXPRESSION CLASSIFICATION WITH HAAR FEATURES, GEOMETRIC FEATURES AND CUBIC BA‰ZIER CURVES", IU-Journal of Electrical & Electronics Engineering, vol. 13, no. 2, 2013, pp.1667-1673.
  47. W. Lirong, Y. Xiaoguang, W. Jianlei, X. Jing and Z. Jian, "Facial Expression Recognition Based on Local Texture Features," in 2011 14th IEEE International Conference on Computational Science and Engineering, 2011, pp. 543-546, doi: 10.1109/CSE.2011.96.
  48. Zhan. YZ, Cheng. KY, Chen. YB, et al., "A New Classifier for Facial Expression Recognition: Fuzzy Buried Markov Model," in J. Comput. Sci. Technol. vol. 25, no. 3, 2010, pp. 641-650. https://doi.org/10.1007/s11390-010-9353-x
  49. N. Devi and M. V. V. R. M. K. Rao, "A novel method to achieve optimization in facial expression recognition using HMM," in 2015 International Conference on Signal Processing and Communication Engineering Systems, 2015, pp. 48- 52.
  50. J. See, "Probabilistic Bayesian network classifier for face recognition in video sequences," in 2011 11th International Conference on Intelligent Systems Design and Applications, 2011, pp. 888-893.
  51. Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik, "Rich feature hierarchies for accurate object detection and semantic segmentation," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014, pp. 580-587.
  52. Ren. S, He. K, Girshick. R, and Sun. J, "Faster R-CNN: Towards real-time object detection with region proposal networks," Advances in neural information processing systems, Vol. 28, 2015.
  53. Huang. L, Yang. Y, Deng. Y, and Yu. Y, "Densebox: Unifying landmark localization with end to end object detection," arXiv preprint arXiv:1509.04874, 2015.
  54. Shuo Yang, Ping Luo, Chen-Change Loy, and Xiaoou Tang, "From facial parts responses to face detection: A deep learning approach," in Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2015, pp. 3676- 3684.
  55. Akshay Asthana, Stefanos Zafeiriou, Shiyang Cheng, and Maja Pantic, "Robust discriminative response map fitting with constrained local models," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013, pp. 3444-3451.
  56. Yi Sun, Xiaogang Wang, and Xiaoou Tang, "Deep convolutional network cascade for facial point detection," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013, pp. 3476-3483.
  57. Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang, "Facial landmark detection by deep multi-task learning," In European conference on computer vision(ECCV), 2014, pp.94-108.
  58. K. Zhang, Z. Zhang, Z. Li and Y. Qiao, "Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks," in IEEE Signal Processing Letters, vol.23, no.10, 2016, pp. 1499-1503, doi: 10.1109/LSP.2016.2603342.
  59. Riza Alp Guler, George Trigeorgis, Epameinondas Antonakos, Patrick Snape, Stefanos Zafeiriou, and Iasonas Kokkinos, "DenseReg: Fully Convolutional Dense Shape Regression In-The-Wild," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 6799-6808
  60. Hu P, and Ramanan D, "Finding tiny faces," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp.951-959.
  61. Wayne Wu, Chen Qian, Shuo Yang, Quan Wang, Yici Cai, and Qiang Zhou, "Look at Boundary: A Boundary-Aware Face Alignment Algorithm," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 2129-2138
  62. Goodfellow. I, Pouget-Abadie. J, Mirza. M, Xu. B, Warde-Farley. D, Ozair. S, ... and Bengio. Y, "Generative adversarial networks," Communications of the ACM, vol. 63, no. 11, 2020, PP.139-144. https://doi.org/10.1145/3422622
  63. Mirza M, and Osindero S, "Conditional Generative Adversarial Nets," arXiv eprints, 2014, arXiv: 1411.1784.
  64. Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel, "InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets," Advances in Neural Information Processing Systems 29 (NIPS),Vol 29,2016.
  65. Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros, "Unpaired image-to-image translation using cycle-consistent adversarial networks," in Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017, pp. 2223-2232.
  66. Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo, "StarGAN: Unified Generative Adversarial Networks for MultiDomain Image-to-Image Translation," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 8789-8797.
  67. SUN X, and DING X, "Data augmentation method based on generative adversarial networks for facial expression recognition sets," Computer Engineering and Applications, vol. 56, no. 4, 2020, pp.115-121.
  68. Rui Huang, Shu Zhang, Tianyu Li, and Ran He, "Beyond Face Rotation: Global and Local Perception GAN for Photorealistic and Identity Preserving Frontal View Synthesis," in Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017, pp. 2439-2448.
  69. Wang W, Sun Q, Fu Y, Chen T, Cao C, Zheng Z, ... ,and Xue X, "Comp-GAN: Compositional generative adversarial network in synthesizing and recognizing facial expression," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 8789-8797
  70. Susskind. JM, Hinton. GE, and Movellan. JR, "Generating facial expressions with deep belief nets," Affective Computing. InTech, 2008, pp.421-440.
  71. Yang R, Zhang YW, and Gou S, "Gabor Features Combined with Deep Belief Networks for Face Recognition," Sensors and Microsystems, vol. 36, no. 5, 2017, pp.68-70.
  72. Huang Shouxi, and Qiu Weigen, "Face expression recognition based on improved deep belief network," Computer Engineering and Design, vol. 38, no. 6, 2017, pp.1580-1584.
  73. Li. C, Wei. W, Wang. J, Tang. W, and Zhao. S, "Face Recognition Based on Deep Belief Network Combined with Center-Symmetric Local Binary Pattern," Advanced Multimedia and Ubiquitous Engineering, vol 393, 2016, pp.277-283. https://doi.org/10.1007/978-981-10-1536-6_37
  74. G. Williams, R. Baxter, Hongxing He, S. Hawkins and Lifang Gu, "A comparative study of RNN for outlier detection in data mining," 2002 IEEE International Conference on Data Mining, 2002, pp. 709-712, doi: 10.1109/ICDM.2002.1184035.
  75. Yang Y, Fang D, and Zhu D, "Facial expression recognition using deep belief network," Rev. Tec. Ing. Univ. Zulia, vol.39, no.2, 2016, pp.384-392.
  76. K. Zhang, Y. Huang, Y. Du and L. Wang, "Facial Expression Recognition Based on Deep Evolutional Spatial-Temporal Networks," in IEEE Transactions on Image Processing, vol. 26, no. 9, 2017, pp. 4193-4203, doi: 10.1109/TIP.2017.2689999.
  77. Chaturvedi, R. Satapathy, S. Cavallari, and E. Cambria, "Fuzzy commonsense reasoning for multimodal sentiment analysis," PatternRecognition Letters, vol.125, 2019, pp.264-270. https://doi.org/10.1016/j.patrec.2019.04.024
  78. S Zhao, H Cai, H Liu, J Zhang, and S Chen, "Feature Selection Mechanism in CNNs for Facial Expression Recognition," in BMVC, 2018, pp. 317.
  79. Yang X and Shang ZH, "Facial expression recognition based on improved AlexNet," Laser & Optoelectronics Progress, vol.57, no.14, 2020, pp.243-250.
  80. Dhankhar. P, "ResNet-50 and VGG-16 for recognizing facial emotions," International Journal of Innovations in Engineering and Technology (IJIET), vol.13, no.4, 2019, pp.126-130.
  81. Cui Ziyue, Pi Jiatian, Chen Yong, Yang Jiezhi, Xian Yan and Wu Zhiyou, "Combining improved VGGNet and focal loss for facial expression recognition," Computer Engineering and Applications, vol.57, no.19, 2021, pp.171-178.
  82. SHI Hao, XING Yuhang, and CHEN Lian, "Facial expression recognition based on multi-scale feature fusion and attention mechanism," Microelectronics & Computer, vol.39, no.3, 2022, pp.34-40.
  83. Shen Hao, Meng Qinghao, and Liu Yinbo, "Face expression recognition based on multilayer feature fusion with lightweight convolutional networks," Advances in Lasers and Optoelectronics, vol.58, no.6, 2021.
  84. Chen Jiamin and Xu Yang, "Expression recognition based on attention splitting convolution residual network," Laser & Optoelectronics Progress, vol. 59, no.18, 2022.
  85. JIANG Yuewu, ZHANG Yujin, and SHI Jianxin, "Expression Recognition Method Combining Key Points and Residual Network of Weight Allocation," Computer Engineering and Applications, vol.58, no.17, 2022, pp.181-188.
  86. Gao Jingwen, Cai Yongxiang, and He Zongyi, "TP-FER: A Three-Channel Face Expression Recognition Method Based on Optimized Convolutional Neural Network," Computer Applications Research, 2021.
  87. Pengbo Yin, Weimin Pan, and Navy Zhang, "A lightweight method for face expression recognition based on convolutional attention," Advances in Lasers and Optoelectronics, vol.58, no.12, 2021.
  88. Liang Huagang and Lei Yixiong, "Enhanced Separable Convolutional Channel Features for Expression Recognition," Computer Engineering and Applications, vol. 58, no.2, 2022, pp.184-192.
  89. Li Zhe, Li Keqing, and Deng Junyong, "Research on face expression recognition method based on multi-feature combination," Computer Application Research, vol.4, 2020.
  90. T. Yu, J. Yu, Z. Yu and D. Tao, "Compositional Attention Networks With TwoStream Fusion for Video Question Answering," in IEEE Transactions on Image Processing, vol. 29, 2020, pp. 1204-1218, doi: 10.1109/TIP.2019.2940677.
  91. K. Zhang, Y. Huang, Y. Du and L. Wang, "Facial Expression Recognition Based on Deep Evolutional Spatial-Temporal Networks," in IEEE Transactions on Image Processing, vol. 26, no. 9, 2017, pp.4193-4203. https://doi.org/10.1109/TIP.2017.2689999
  92. Liu C, Tang T, Lv K, and Wang M, "Multi-feature based emotion recognition for video clips," in Proceedings of the 20th ACM International Conference on Multimodal Interaction, 2018, pp. 630-634.
  93. D. H. Kim, W. J. Baddar, J. Jang and Y. M. Ro, "Multi-Objective Based SpatioTemporal Feature Representation Learning Robust to Expression Intensity Variations for Facial Expression Recognition," in IEEE Transactions on Affective Computing, vol.10, no.2, 2019, pp. 223-236, doi: 10.1109/TAFFC.2017.2695999.