FlowSE: Efficient and High-Quality Speech Enhancement via Flow Matching

Generative models have excelled in audio tasks using frameworks like codec, language models, diffusion, and flow matching. However, existing generative approaches for speech enhancement (SE) face notable challenges: language model-based methods often degrade timbre and intelligibility due to quantization loss, while diffusion models suffer from complex training and high inference latency. To address these challenges, we propose FlowSE, to our knowledge, the first flow-matching model for SE. FlowSE trains on noisy mel spectrograms and optional character sequences, optimizing a condition flow matching loss with ground-truth mel spectrograms as labels. It implicitly learns speech’s temporal-spectral structure and text-speech alignment. During inference, FlowSE can operate with only noisy mel spectrograms or with additional transcripts, achieving impressive results in both scenarios. With transcripts, FlowSE delivers even higher performance. Extensive experiments demonstrate that FlowSE significantly outperforms state-of-the-art generative methods, establishing a new paradigm for generative-based SE and demonstrating the potential of flow matching to advance the field. Our code, pre-trained checkpoints, and audio samples are available