Voice notes with transcripts: how I built them

The gesture

Recording starts with a press on the microphone beside the message box. You talk, you let go, and the note is sent. I deliberately avoided a separate recording screen with preview and confirm steps, because every extra step turns a quick note into a small chore. If you change your mind, you can delete the note afterwards.

From audio to text

After a note is sent, three things happen in order.

• The audio is stored with the message and a waveform is shown for it.
• A speech recognition step on the server turns the audio into text. Its status moves from queued to processing to done, or to failed or skipped.
• The transcript appears next to the audio as text attached to the message.

Why the transcript is attached to the message

A transcript on a separate screen is a transcript people forget about. I attach it to the message so it appears where people already look. The audio is not replaced; it stays for tone and for when listening is simpler.

Trade-offs I am weighing

Speech recognition is imperfect, especially with noise, accents, names and several languages at once. A transcript is machine generated, and the audio is always one press away so you can check it. The language hint comes from your locale, and I do not promise accuracy in any particular language.

Transcripts of live voice rooms are stored too, with the speaker, start and end time and text of each segment. People who can view and read the history of that channel can read them, and how long they are kept depends on the plan: 30 days on the free plan.

Why not transcribe everything the same way

For a voice note, a short delay between recording and transcript is acceptable, and it lets the system work on the whole clip at once, which usually gives better results than word-by-word guessing. For a live room the delay matters more, and the trade-off between speed and accuracy is different. I treat the two as separate problems.

This is one reason voice notes and room transcripts are described separately. They look similar on screen and behave differently underneath: a note gets one transcript after it is sent, while a room produces a stream of short segments, each with a speaker and a time.

What the transcript should never be

A transcript should never be treated as a verdict. It can be wrong, and a person should not be held to words that a machine put in their mouth. The audio remains the source, and the transcript is labelled as machine generated.

A transcript can also fail or be skipped, and then the note simply plays as a voice note. If you have a concern about how your voice data is handled, write to [email protected] and I will answer it.

A note on accessibility

A transcript is also useful for people who are deaf or hard of hearing, and for anyone in a place where audio is not an option. That use is part of why I want transcripts to sit next to the recording and not behind a menu. It is also why they have to be readable: short paragraphs, correct punctuation where the system can manage it, and a clear mark where it is unsure. I do not have a quality target to quote, and I will not make one up.

What is next

Voice notes are live, so the best feedback is from people using them. Sign up free at yappio.chat/register, hold the microphone button next to the message box and tell us what is wrong at [email protected].