AI and transcription
what picks the moments, and what writes down the words.
Two console sections decide how a source is analysed: Transcription turns speech into words with timings, and AI lets Claude pick the moments. Transcription is required for analysis; the AI key is optional.
01What works without which key
| You have | What happens |
|---|---|
| No key, no whisper.cpp | Uploads, the editor's timeline and layout, rendering and exports work. A source cannot be analysed: the console check "Transcription is set up" says so, and clips can only be made by hand without captions. |
| whisper.cpp only (no key at all) | Full flow with no API cost: local transcription, moments picked by the built-in picker. |
| A transcription key, no Anthropic key | Transcription by the provider; moments by the built-in picker from energy, speech rate, punctuation and chat (packages/moments/src/heuristic.ts). The picker still gives each clip a hook line (when the clip has a short complete sentence that fits), up to three post titles and hashtags, all taken from the clip's own words. A Claude key adds reasons written by AI, better titles, the complete-thought check and retitling. |
| Transcription and an Anthropic key | Claude picks the moments, checks each clip is a complete thought, writes titles, hooks, hashtags and post texts, and learns a workspace's preferences from its approvals and rejections. |
The console check "Anthropic key is set" is a warning, not a blocker, for this reason.
02AI (Anthropic)
- Get a key At console.anthropic.com create an API key. It starts with
sk-ant-. Add billing there; Anthropic bills you for every call. - Paste it Console, System, Integrations, Anthropic API key. Use the "Test" button.
- Pick the models Console, Setup, AI.
| Field | Default | Meaning |
|---|---|---|
| Model | Claude Opus 5.5 (best) | The main model. Also offered: Claude Sonnet 5.5 (faster, cheaper), Claude Haiku 5.5 (cheapest). |
| Model for short jobs (titles, rewrites, translations) | Claude Opus 5.5 | A smaller model here lowers your cost for the small actions. |
| Model for picking moments | Same as the main model | The largest call: the whole annotated transcript of a source. |
| Model for the complete-thought check | Same as the main model | Runs once per analysis when "Check every suggested clip for a complete thought" is on. |
| Retry declined requests on a fallback model | On | |
| Workspaces may bring their own Anthropic key | Off | On: a workspace owner can enter their own key on the app's Workspace page. That workspace then pays its own AI usage, and AI moment picking costs it no credits. |
How Claude is used: it reads the transcript with markers for energy, laughter, pauses and chat spikes, and answers with sentence numbers. The server, not the model, decides every cut, and snaps it to sentence boundaries (packages/moments/README.md). If the model fails, the built-in picker takes over.
The console's AI usage page lists every call with its model and tokens. Check your provider's dashboard for what it costs you, and set the credit prices to cover it (Credits).
03Transcription: choose a provider
Console, Setup, Transcription, Provider:
| Provider | Needs | Speaker labels | Notes |
|---|---|---|---|
| ElevenLabs Scribe (default) | ElevenLabs API key (Integrations) | Yes, with "Tell speakers apart" on | One request per file. Speaker turns also help reframing follow the person who talks. |
| Whisper over HTTP | Whisper API key (Integrations), base URL, model | No | Any OpenAI-compatible server with /audio/transcriptions and word timestamps. Long audio is cut into pieces at pauses and merged. |
| whisper.cpp on this server | The program and a model file on the server | No | No API cost. Uses the server's CPU. |
04ElevenLabs Scribe
- Get a key At elevenlabs.io, API keys.
- Paste it Console, System, Integrations, ElevenLabs API key.
- Provider Console, Setup, Transcription: "ElevenLabs Scribe (speaker labels)". Leave "Tell speakers apart (ElevenLabs)" on for podcasts and interviews.
05Whisper over HTTP (Groq, OpenAI or compatible)
- Get a key From Groq, OpenAI, or your own compatible server.
- Paste it Console, System, Integrations, Whisper API key.
- Provider Console, Setup, Transcription: "Whisper over HTTP", then the two fields below.
| Service | Whisper API base URL | Whisper model |
|---|---|---|
| Groq (the defaults) | https://api.groq.com/openai/v1 | whisper-large-v3-turbo |
| OpenAI | https://api.openai.com/v1 | whisper-1 |
These services cap one upload at 25 MB, so the server sends audio in pieces of about ten minutes when a file is larger (packages/transcribe/src/whisper-http.ts).
06Local whisper.cpp: install and download a model
whisper.cpp is a separate open-source program (MIT licence) that is not part of this package. You build it once on the server. The commands below follow its own README at github.com/ggml-org/whisper.cpp; if they change there, the README wins.
sudo apt install -y git cmake build-essential
sudo git clone https://github.com/ggml-org/whisper.cpp.git /opt/whisper.cpp
cd /opt/whisper.cpp
sudo cmake -B build
sudo cmake --build build -j --config Release
# download a model (tiny, base, small, medium, large-v3 …)
sudo sh ./models/download-ggml-model.sh small
# the two paths the console asks for
ls /opt/whisper.cpp/build/bin/whisper-cli
ls /opt/whisper.cpp/models/ggml-small.binThen in the console, Setup, Transcription:
| Field | Value in this example |
|---|---|
| Provider | whisper.cpp on this server |
| Path of whisper-cli | /opt/whisper.cpp/build/bin/whisper-cli |
| Path of the whisper.cpp model file | /opt/whisper.cpp/models/ggml-small.bin |
The status check "Transcription is set up" turns green when both files exist. The user that runs Vodmill must be able to read and run them.
Larger models are more accurate and slower, and need more memory. The model must be a multilingual one (no .en in the name) if your sources are not English. Limits noted in packages/transcribe/README.md: word times from whisper.cpp are approximate, and languages written without spaces come back as one word per segment, so word-by-word captions suit them poorly.
07Languages and spelling
- The language of a source is detected by the provider unless the user sets it.
- Each workspace has "Words to spell right" on its Clip settings page: one per line,
what is heard = how it is written. They are applied to captions and titles. - Captions are drawn with the bundled fonts, which cover Latin and extended Latin scripts. Other scripts fall back to a system font on the server (
packages/captions/README.md).