MoE Emotion Predictor: Real-Time Japanese Emotion Recognition

Real-time Japanese emotion recognition from text, audio, or both, using a Mixture of Experts architecture with learned soft gating. Given Japanese speech or text, the system predicts a discrete emotion label and continuous valence/arousal scores — live, from browser mic input streamed over WebSocket, or from an uploaded audio file.

The text expert encodes input with cl-tohoku/bert-base-japanese-v3 (768-dim); the audio expert uses facebook/wav2vec2-base (768-dim) to capture prosodic features without transcription. A learned MoE router dynamically weights both modalities at inference time, degrading cleanly when only one is available.

Two output heads run in parallel: a 7-class softmax classifier (happy, sad, angry, neutral, anxious, surprised, disgusted) and a regression head for continuous valence and arousal on a 1–9 scale.