Multimodal Face Expression Recognition Using Parametric Exponential Linear Unit-Long Short-Term Memory

Abstract

Multimodal facial expression recognition combines information from multiple modalities of audio and video to achieve the required accuracy and robustness. By integrating different data sources, multimodal systems capture different aspects of human expression. However, accurately recognizing facial expressions across audio and video modalities causes challenges due to variations in expression representation. In this research, Parametric Exponential Linear Unit-Long Short-Term Memory (PELU-LSTM) is proposed to accurately recognize multimodal facial expressions. Initially, the SAVEE dataset is used to evaluate the performance of the proposed method which contains audio and video frames. In audio pre-processing, a wiener filter is deployed to minimize background noise, while a Gaussian Weighting Function (GWF) is employed to aggregate the entire video into a smaller number of frames which also minimizes the information loss. The Mel-Frequency Cepstral Coefficient (MFCC) is utilized to extract audio features, while the Histogram of Gradient (HOG) and Local Binary Pattern (LBP) are employed for extracting the video features. Then, concatenation is performed to fuse a single feature vector. Finally, PELU-LSTM recognizes the facial emotional expressions accurately. The proposed technique achieves a high accuracy of 99.75%, as compared to the existing techniques like Bi-directional LSTM-Convolution Neural Networks (Bi-LSTM-CNN), attention-based 2D CNN with LSTM and K-means clustering-based Kernel Canonical Correlation Analysis (KMKCCA)