Project Details
Mendwave — AI Speech Editing 2026
Project Overview
Mendwave lets someone fix a finished voice recording without re-recording it. The user edits the transcript; the system diffs the new text against the original, locates the changed spans by forced alignment, regenerates only those spans in the original performer's voice, verifies each candidate against speaker-identity and content-accuracy gates, and splices the result back at low-energy cut points. Everything outside the edit is preserved sample for sample.
The architecture stores no voiceprints — the voice is conditioned in-context from the surrounding audio in the same session. That makes the system usable in legal and contractual settings where voice cloning is prohibited outright, which is the structural difference from clone-based tools that require a pre-trained model of the speaker.
Built end to end and solo: research, model training, inference pipeline, evaluation harness, web product, and billing — shipped to paying customers.
The Problem
Re-recording a single misspoken word is expensive out of all proportion to the fix. In post-production it means an ADR or pickup session: booking talent, matching the original mic and room, matching energy and pacing. Often it is not just expensive but impossible — the talent is unavailable, the contract forbids synthetic replacement, or the original acoustic conditions can't be reproduced.
Voice cloning fails for a different reason: it requires storing a persistent voice model, which triggers biometric-data regimes, SAG-AFTRA digital-replica provisions, and studio procurement objections. The wedge is that “I have a recording with a problem” is a fundamentally different product from “I need a voice from scratch”. The second is well served; the first was not.
The Edit Pipeline
Transcript edit to spliced master
- Separation (opt-in): BS-RoFormer voice/background separation for mixed audio, with the music-and-effects bed re-laid under the edited voice using crossfade-looped fills — never time-stretched.
- Alignment: Forced alignment (CrisperWhisper / faster-whisper) produces word-level timestamps, cached by content hash at roughly 60× realtime on a 4090.
- Edit operations: Word-level diff of original vs. corrected transcript yields edit spans; spans within two gap words are merged, expanded by a context word on each side, and cut points are snapped to local energy minima.
- Generation: Windowed in-context regeneration over ±10 s of word-snapped context, with best-of-N sampling.
- Verification: Two hard gates — speaker similarity by equal-length embedding cosine, and margin-aware WER on re-transcription — plus prosody, loudness, and seam diagnostics.
- Stitching: LUFS normalization, room-tone matching, and equal-power crossfades at the seam boundaries.
Two invariants
- Edit-region-only. The master is sacred. Audio outside the splice is bit-for-bit the original file, and every quality metric scores the edit region alone.
- Gate-before-promote. Nothing is spliced unless it clears the gates. Borderline candidates route to human review rather than auto-applying; filters triage but never silently reject.
The Model — MendEdit
The first production generation stage used SSR-Speech, an autoregressive neural-codec LM in the VoiceCraft lineage at 16 kHz. It worked on calm read speech and failed structurally on expressive material: with no explicit duration model for the infill region, it regresses to mean speaking rate. That diagnosis triggered the rebuild.
MendEdit is a CosyEdit2-style reproduction on CosyVoice2-0.5B at 24 kHz, built as a staged sequence with hard gates between stages.
- Synthetic edit-triplet data pipeline over a licensed speech corpus, with expressive-intensity-stratified sampling so 40%+ of training edits fall in high-arousal regions — a deliberate divergence from the source paper, aimed at the exact failure mode.
- In-context sequence formatting and dataloader with ±0.12 s seam-margin loss masking.
- Stage-A fine-tune of the Qwen2LM 0.5B backbone, then Stage-B fine-tune of the GOT-CFM conditional flow-matching decoder.
- Inference integration behind an environment-variable swap, requiring no orchestrator changes, followed by bench evaluation stratified by arousal bucket and scored on the edit region.
- Optional per-speaker LoRA adapters, shipping only with a consent manifest and a verified cryptographic deletion path — because a LoRA is a stored voice model.
Research arms built and evaluated in parallel: prosody-conditioned generation via an additive zero-init Conv1d projection injecting per-frame F0/energy contours into the flow decoder; non-autoregressive fixed-duration masked infill (PlayDiffusion / F5-TTS family) under a substitution-only word-count constraint; and a signal-domain WORLD/PSOLA prosody transplant as the cheap baseline.
Evaluation Harness
Built before the model work, and the reason the model work was correctly directed. It programmatically injects roughly 100 known edits across held-out material — single-word substitutions, multi-word phrases, full sentences, insertions, deletions — with ground truth known, producing a scorecard broken out by edit type, edit length, and arousal bucket.
Two calibration tests run on every pass: candidates that are deliberately a different speaker, to confirm the speaker gate rejects rather than merely passes, and candidates that say deliberately wrong words, to confirm the content gate rejects. Passing clean candidates proves nothing about a gate; rejecting bad ones does.
Findings
Whole-clip scoring is a lie
Standard speech-editing benchmarks score quality across the entire clip, averaging the edited region together with untouched original audio and masking bad edits. Re-scoring on the edit region only inverted the model rankings and revealed that the speaker-similarity filter had been silently accepting broken audio. Edit-region-only scoring became a project-wide invariant.
The speaker-verification metric was measuring nothing
On source-separated film audio, ECAPA scored the same real speaker against himself at 0.122 — statistically indistinguishable from two different people (~0.13). WavLM-TDNN scored the same pair at 0.915. Re-scoring every film edit under WavLM put the real synthesis deficit at about 0.05, meaning nearly the entire apparent identity cliff was measurement contamination, not model failure. The scoring standard was frozen: WavLM-TDNN for separated or degraded audio, ECAPA valid only on clean.
Acoustic domain shift, quantified
Speaker identity held at ~0.94 on clean in-domain speech and collapsed to ~0.43–0.45 on real film/TV dialogue. Root cause: training corpora are anechoic and close-mic'd, while production dialogue carries room acoustics, boom/lav perspective, and a post chain baked into the signal. Fix path built as RIR convolution → microphone impulse responses → randomized EQ/compression, calibrated against measured channel statistics from real film dialogue.
Failure mechanism isolated to a specific number
Film-context conditioning caused the model to overshoot target duration by ~1.84×, tripping a hard length gate and dropping candidate yield to 49% of the clean-context rate, while game-voice context retained 94%. Naming the mechanism as duration overshoot rather than generic quality degradation made it addressable.
A feature that looked helpful was catastrophic
An augmented-context mode collapsed content fidelity from 79.1% pass@N to 3.9%. The bench gave a clean causal split — the feature engaged on 97.4% of candidates, and engaged candidates passed at 1.3% vs. 50% for skipped ones — with dual-ASR agreement confirming it wasn't a transcription artifact. Default locked off, decision recorded in the experiment ledger.
Gate calibration, empirically
Of the candidate quality signals, tempo deviation was the strongest failure predictor (AUC 0.783), followed by speaker similarity (0.708); several intuitively appealing signals — background continuity, seam spectral flux — failed the bar and were not promoted. A goodness-of-pronunciation gate cleared spec at 93% catch / 10% false-reject, while a phoneme-error-rate gate managed only 35% catch.
Production quality
~87.5% of edits auto-apply after gating with 0% survived-WER on passing edits; 12.5% route to human review, concentrated in exactly the long-span cases the architecture is known to struggle with. Bench median is 5.2% WER on gated output, with zero edits above 20% WER shipping unflagged.
The Product
Frontend
- React SPA with a project/recording library
- Block-segmented transcript editor keyed to word-level time ranges
- Waveform view and before/after A/B playback
- Per-edit confidence report, accept/reject with reason capture
Backend & Billing
- FastAPI on Modal — A10G GPU engine, warm-class scaledown
- CPU web tier for auth, jobstore, billing, analytics, ETA
- Supabase Postgres for accounts, credit ledger, event stream
- Stripe Checkout with webhook-credited balances
Consent Layer
- Rights attestation bound to the audio's SHA-256 content hash — never a speaker embedding
- Performer magic-link sign-off gating professional use, enforced server-side at submit
- C2PA-style provenance manifest on output
Data Separation
- Transcript text, edit words, audio, and human verdicts stay in the private results DB
- Analytics receives only counts, durations, categories, and outcomes
- A denylist scrubber strips content and PII before anything crosses the boundary
Legal & Data Governance
- All training data tagged research_only vs. commercial_ok at ingestion as a hard boundary — research-only corpora restricted to calibration and evaluation, never to a shippable checkpoint.
- Established through case-law research that copyright liability for training data attaches at download, not deployment, and that model taint from infringing data does not wash out through subsequent fine-tuning — retraining from scratch is the only remedy. A proposed “train on scraped audio now, swap in licensed data later” plan was rejected on those grounds.
- Voice and personality rights treated as a separate regime from copyright (ELVIS Act, California AB 2602, NO FAKES Act, SAG-AFTRA agreements).
- Consent architecture designed to make only true claims: proof-of-provenance, proof-of-localization, match-evidence, and consent-on-record — explicitly not proof-of-voice, since an attestation can't verify identity at the acoustic level.
Business
- Live in production with real payments — Stripe Checkout at $5 per fix, with purchase and refund verified end to end, and the first paying customers acquired.
- Pricing designed as a discovery instrument: a single flat price open to all segments, so conversion patterns reveal the real beachhead through revealed preference rather than interview claims.
- 18-month bottom-up revenue model with named disconfirming risks, and unit economics with GPU COGS modeled per edit.
- Submitted to Y Combinator (Fall 2026); recruited a technical co-founder and wrote a full engineering onboarding document covering architecture, infrastructure, working agreements, and access.
- Market and competitive research across five segments — film/TV post, game localization, dubbing, audiobooks, e-learning — grounded in SAG-AFTRA AI provisions and EU AI Act Article 50 transparency requirements.
Strategic evolution
The target segment moved with evidence — from audiobook narration, to film/TV ADR, and most recently to automated QC and repair of AI-generated speech at catalog scale — once research established the governing criterion: the product is only worth money where no one can re-record for free. That criterion eliminated human-narrated audiobooks, union dubbing, and voice agents. The pivots were driven by falsification, not fashion.
Project Information
Founder & ML Engineer
Speech ML Research / Production Product
CosyVoice2-0.5B, Qwen2LM, conditional flow matching (GOT-CFM), SSR-Speech, F5-TTS, PlayDiffusion, CrisperWhisper / WhisperX forced alignment, BS-RoFormer & HTDemucs separation, pyannote, ECAPA-TDNN / WavLM-TDNN / CAMPPlus, emotion2vec, LoRA/PEFT, WORLD/PSOLA
Python, PyTorch, FastAPI, React, TypeScript, SQLite, Postgres, ffmpeg, pedalboard, librosa
Modal (serverless GPU), RunPod A100-80GB, vast.ai, Vercel, Supabase, Stripe, Docker
Staged builds with DONE-WHEN gates, experiment ledger with disconfirming evidence, observe-mode gating before enforcement, content/metadata trust boundaries, C2PA provenance