Time accurate speech transcription is essential in critical applications, such as medical conversation transcription, reading assessment, and atypical speech recognition. Encoder-decoder automatic speech recognition models, such as Whisper, do not produce word-level timestamps out-of-the-box for all languages. Existing approaches to word-level time stamp estimation either use an external CTC model capable of frame level phoneme/character prediction which is computationally expensive, or use the cross attention scores internal to the model which gives coarse time stamps. We consider timestamped speech transcription for Hindi in this paper. We propose a CTC-based approach, in which we train a low-complexity character level CTC head directly on the Whisper encoder output, to address the complexity and granularity limitations of existing works. We evaluate the system using manually-annotated word-level timestamps. Experiments show that the proposed internal-CTC based approach performs similar to using external CTC model and better than the cross attention score-based approaches. Further, we study the usage of CTC-head output to detect and mitigate hallucinations in conversational speech. Our results show that a simple token length criterion derived from the CTC output can improve the transcription quality for short segments where hallucinations are more probable.
The timings are uncalibrated. CTC boundaries are systematically tight: words start about 38 ms late and end about 100 ms early relative to human annotation, so the model's span sits inside the human's on almost every word. Subtracting those two constants cuts mean boundary error from 73.0 ms to 42.0 ms. Nothing on this page has been corrected, so what you see and hear is the raw output.