Speech Emotion Recognition

Speech carries more than just words. The way it sounds, with changes in pitch, energy, rhythm, and tone, can give listeners hints about how an utterance feels emotionally. This is the focus of the Speech Emotion Recognition project, which uses machine learning to spot these patterns in audio and give it a label describing the emotion.

The project relies on a dataset called RAVDESS, which contains acted speech recordings expressing different emotions. There are six possible labels: neutral, happy, sad, angry, fearful, and disgust. Since the speakers are acting, the system's output should be seen as a prediction based on the patterns in this dataset, rather than a reliable reading of a person's true emotional state.

Before a model can classify an audio clip, it needs to be prepared. The audio is converted into a consistent form, with mono, resampling to 22,050 Hz, trimming of silence, and adjustment to three seconds. Normalization of the waveform is also done. Two types of audio features are prepared: Mel-frequency cepstral coefficients (MFCCs) and a log-mel spectrogram.

MFCCs use 40 coefficients, each with a time-based mean and standard deviation, creating an 80-value feature vector. The spectrogram shows how sound energy changes over time across frequency bands, and is turned into a 128-by-128 image for a convolutional neural network.

Several modeling approaches are compared. For the MFCC training, a support vector machine, a random forest, and an XGBoost classifier are evaluated on validation and test data. The best model is chosen using validation macro-F1, which measures performance across all classes. A separate workflow trains a convolutional neural network on the log-mel spectrograms, using validation accuracy to guide learning-rate adjustments and early stopping.

The CNN is saved as its own model. The API currently uses the MFCC pipeline, while the CNN training workflow is available in the repository but not loaded or served by the API.

To ensure the evaluation is speaker-independent, the data loader assigns each actor's recordings to one of three splits: training, validation, or test. Each of these splits is 15% of the data. This helps reduce the chance that the same person's voice appears in both training and evaluation data, making the evaluation more meaningful.

The API provides an HTTP endpoint for audio classification, accepting WAV, MP3, FLAC, OGG, and M4A files up to 20 MB. It gives the predicted label, top three class probabilities, clip duration, and a disclaimer. The endpoint also has health and interactive API documentation.