Hallucinations in audio transcriptions

Just a short follow on from my litter data post. As I mentioned, I’m recording myself wandering the streets of West Lothian and narrating what I pick up to put in my bag. I then save a transcript of the recording, give it to an AI which produces me a list of items, highlighting any where the transcript was garbled. I then manually correct, and run it back through the AI and it adds it to my litter master-spreadsheet. It’s a slow process and I’m looking at ways to speed it up and hence why I started to look at the use of Buzz and different versions of Whisper.ai, to see if it could improve the speed. In my last post, I’d started tinkering with the tiny and medium Whisper models and had seen a big improvement over the transcription generated by the Apple Voice Memos application.

One other issue with the Apple transcripts is that it has no time indicator within the transcript – so if something needs to be checked by a human, there’s no easy way to do this. I tried to count the litter as I walked but it was too easy to forget where I was, so that didn’t work, hence another reason to look at other transcription technologies.

I’ve done a bit of testing. Earlier this week I did a brand new litter pick. I used the same settings and same prompt for all of the transcription request and just varied the model. I took the transcript, as usual, from the Apple Voice Memo app but also ran it through three Whisper models – Small, Medium and Large Turbo. With Large Turbo, it took much longer so I assumed it would be more accurate. But the results were not what I expected at all.

So, what happened?

I’ve got a spreadsheet which has 3 columns, one per model, and a row for each item I found on my litter pick. I manually walked through the recording noting down the correct items where the models got them wrong. Large Turbo added a significant number of random new rows that didn’t exist in the original recording. When I picked through the data, I found that the Medium and Small models also did this but at a much smaller scale.

I had collected 355 items on the pick.

Model	Number of items missed	Number of hallucinations
Small	37	33
Medium.en	115	15
Large Turbo	18	130
Apple Voice Memo Transcription	4	0

The quality of the audio is good. I relistened to it and there was nothing that was hard to hear or not understandable.

A few things to note:

• Medium.en just failed to give any transcription towards the last third of the audio recording, it was just blank. That was the reason for the vast majority (110) of the items missed.
• Most of the hallucinations occurred in all models when there was a gap ie where there was a bit of time between reporting one item and another.
• Although the Apple Voice Memo Transcription didn’t hallucinate, it’s accuracy was poorer than all the others and it was much harder to make an educated guess as to what a specific item would be.

I got AI to do a bit of analysis on the quality of the transcript in terms of the item names asking it to sort into Accurate, Recognisable (ie you can guess what the transcript meant), Not recognisable (ie you can’t guess what the transcript meant) and Missed completely (where the item is missing from the transcript). I then checked AI’s categorisation, it was slightly more generous on the ‘recognisable’ tag than I would have been but mostly it concurred with my thoughts. It found:

So, ironically and a bit annoyingly… the Apple Voice Memo Transcription sort of wins. An AI could correct about 97% of the items from the transcript, there were only 4 it completely missed and it didn’t hallucinate at all. I’m still confused about how it could miss any, but I will do some investigation into that. The annoying thing for me is that the 3% that need human checking are hard to find due to the lack of timestamps in the transcript file. The least accurate was Medium at 67% – massively reduced by the fact that it missed almost one third of the transcript.

If we look at the first 57 mins of the transcripts, before Medium.en seemed to die, it was on a par with Apple in terms of recognisability, 97%. Although on a par with Apple in terms of recognisability, Medium still had 15 hallucinations in that time and completely missed 5 (whereas Apple only completely missed 3).

Clearly, for this sort of task, accuracy is important and counting extra items that don’t actually exist or missing items is not really acceptable. I did a bit of a Google and it turns out that the internet is full of people saying the same as me – Whisper hallucinates, the bigger models hallucinate more and it’s typically when there are gaps in speech.

These posts have also suggested a number of solutions. So next steps for me:

• Rerun this pick through medium.en model again to see if it still collapses at 57 mins – I suspect it won’t.
• Try a few of the suggestions in some of these posts including pre-processing to remove non speech gaps, changing the temperature of the whisper.ai model and various other things.

Will do this and report back!

In the meantime – my latest count dashboard is available if you want to see which brand is my current top…