AgenticASR Corrects Speech-to-Text Errors Interactively

A collaborative research team from Shanghai Jiao Tong University, Zhejiang University, Fudan University, and Microsoft spinoff Xiaoice has introduced AgenticASR. This new workflow addresses a common limitation in speech-to-text systems, which typically transcribe audio in a single pass without a mechanism for real-time corrections. By pairing an automatic speech recognition engine with a large language model, the system allows users to edit and refine transcriptions through conversational turns.

The architecture utilizes Qwen3-ASR-1.7B to generate an initial transcription, which is then refined by a Qwen3-32B language model. Instead of rewriting the entire text from scratch—a process that often introduces new errors—the language model categorizes user inputs into confirmation, new input, or correction. When a correction is detected, the model acts as an editor by locating the specific error, understanding the requested change, and applying the fix.

To evaluate the system, the researchers simulated a user interacting over 10 turns and measured performance using a metric called S²ER, which tracks semantic errors. On the GigaSpeech benchmark, AgenticASR reduced S²ER from 21.5 percent at turn 0 to 3.5 percent after 10 turns, while the word error rate fell from 11.9 percent to 10.4 percent. On the AISHELL-NER dataset, which features numerous proper nouns, the semantic error rate dropped from 19.9 percent to 2.0 percent, and the named-entity error rate was halved from 2.4 percent to 1.2 percent. On the mixed-language ASRU2019 benchmark, the system cut S²ER from 28.6 percent to 1.4 percent and lowered the mixed error rate from 6.6 percent to 3.3 percent.

For AI practitioners, this multi-turn refinement workflow offers a robust blueprint for building more reliable voice user interfaces. By decomposing the correction process into discrete steps of identification, comprehension, and modification, developers can prevent LLMs from hallucinating new errors during edits. This approach is highly transferable, providing a potential framework for other interactive AI applications such as collaborative document editing, code review, and iterative design tools.