Teaching a Language Model to Speak

Chapter 0 — Why is speech hard for an LLM?

You know transformers: a sequence of discrete tokens goes in, next-token prediction comes out. Text has a natural tokenization (BPE): a sentence is ~15 tokens per second of reading. Audio is different in two brutal ways:

• Volume. Raw audio is 16,000–44,100 numbers per second. Treat each sample as a token and even self-attention on 10 seconds is intractable (O(n²)), and one second = 16,000 "tokens".
• Structure. Models trained directly on waveforms (WaveNet-era) learn locally plausible sound but globally meaningless output — "babbling speech". They never see abstractions like words, speaker identity, or prosody over a sentence.

The whole field's answer, in one sentence: compress audio into discrete tokens at a manageable frame rate, then treat speech generation as pure language modeling — predict next token, then decode tokens back to a waveform. The two papers you gave me supply the two halves of that pipeline:

• SoundStream — how to turn a waveform into discrete tokens and back (the audio "tokenizer + detokenizer").
• AudioLM — how a transformer LM can generate coherent speech in that token space (the audio "GPT moment").
• Gemma 4 E4B + grafted speech head — a modern, small multimodal LLM, what it can already hear, and how researchers bolt a speaking head onto it.

Read the chapters in order — each builds on the previous. Every chapter ends with a self-check quiz.

Chapter 1 — SoundStream: the audio tokenizer

Research Digest: SoundStream — An End-to-End Neural Audio Codec

Paper: SoundStream: An End-to-End Neural Audio Codec

Authors: Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, Marco Tagliasacchi (Google)

Year: 2021 (arXiv:2107.03312; published in IEEE/ACM TASLP, Nov 2021)

---

1. What problem does it solve?

Traditional audio codecs come in two flavors, and both fail at low bitrates:

• Waveform codecs (e.g., Opus, MDCT-based): transform the waveform to time–frequency, quantize the coefficients, invert at the decoder. Content-agnostic and excellent at medium/high bitrates, but produce audible artifacts when starved of bits.
• Parametric codecs (e.g., CELP, EVS): assume the input is speech, estimate parameters of a hand-crafted synthesis model, quantize those. Better at low bitrates, but only for speech, and quality is capped by the hand-designed parametric model.

SoundStream replaces the entire hand-engineered encode→quantize→synthesize pipeline with a single neural network trained end-to-end. It compresses speech, music, and general audio at 3–18 kbps, runs in real time on a smartphone CPU, is streamable (causal, low latency), and — uniquely — one single model covers all bitrates in that range via quantizer dropout. Headline result: SoundStream at 3 kbps beats Opus at 12 kbps and roughly matches EVS at 9.6 kbps in blind listening tests — i.e., 3–4× fewer bits for equal quality.

2. Architecture, from first principles

The reader is assumed to know transformers; the key conceptual shift is that this is a pure convolutional autoencoder + discrete bottleneck + GAN loss, no attention at all, because it must run causally in real time on audio samples.

2.1 Setup and notation

Input: single-channel waveform x ∈ ℝᵀ sampled at f_s = 24 kHz. The model is G(x) = dec(Q(enc(x))): encoder → quantizer → decoder, producing reconstruction x̂. Everything is causal: convolutions are padded only on the past, so the model can stream and its latency is determined solely by the temporal down/up-sampling ratio.

2.2 Encoder (SEANet-style, fully convolutional)

• One 1D conv (C_enc channels) → B_enc = 4 convolution blocks, each block = 3 residual units with dilated convs (dilations 1, 3, 9) followed by a strided conv that downsamples and doubles channels.
• Strides (2, 4, 5, 8) → total striding factor M = 320 samples per embedding. At 24 kHz that is 75 embeddings per second (one per 13.3 ms) — this frame rate fixes the bit budget math below.
• Final 1D conv (kernel 3, stride 1) projects to embedding dimension D (in practice the bottleneck width is the channel count of the last block, 128·C/32 class designs; the paper's default model has C_enc = C_dec = 32 channels, 8.4 M parameters total).
• Activation: ELU; no normalization layers.

So enc(x) ∈ ℝ^{S×D} with S = T/320.

2.3 Residual Vector Quantizer (RVQ) — the core mechanism

Why not a single codebook? To hit 6 kbps with 75 frames/s, each frame gets r = 6000/75 = 80 bits. A single VQ codebook would need N = 2^80 entries — impossible.

Residual (multi-stage) VQ cascades N_q VQ layers, each with a small codebook of N vectors:

ŷ ← 0;  residual ← y
for i = 1..N_q:
    ŷ        += Q_i(residual)      # quantize the current residual
    residual -= Q_i(residual)      # pass on what's left
return ŷ

Each layer gets an equal slice of the budget: r_i = r/N_q = log₂ N. For 6 kbps with N_q = 8: each codebook has N = 2^(80/8) = 1024 entries. The sum of the layer outputs progressively refines the estimate while keeping shape S×D constant — crucial for bitrate scalability. Bits per frame = N_q · log₂ N.

Codebook training details (all learned end-to-end by backprop):

• EMA updates of codebook vectors (VQ-VAE-2 style; the straight-through estimator carries gradients to the encoder/decoder).
• k-means initialization on the first training batch (instead of random) to improve codebook usage.
• Dead-code revival: track EMA (decay 0.99) of assignments per vector; if a vector's count drops below 2, reassign it to a randomly sampled input frame.

Quantizer dropout → one model, many bitrates. During training, for each example sample n_q ~ Uniform[1, N_q] and use only the first n_q quantizers (structured dropout over quantizer layers). At inference, pick n_q for the target bitrate. Because embeddings keep the same shape regardless of n_q, no architectural change or retraining is needed — unlike product quantization (wav2vec 2.0) or concatenated VQ outputs (Lyra), which require retraining per rate. Bonus: quantizer dropout acts as a regularizer — the scalable model marginally outperformed bitrate-specific models at 9 and 12 kbps.

2.4 Decoder

Mirror of the encoder: 1D conv → 4 blocks, each = transposed (upsampling) conv + the same 3 dilated residual units, strides reversed (8, 5, 4, 2), channels halved at each upsample, final conv (kernel 7, stride 1, 1 filter) projects back to a 24 kHz waveform. Asymmetric capacities were ablated: a lighter encoder barely hurts quality (ViSQOL 3.96 → 3.94) while a lighter decoder hurts more (→ 3.84) — echoing asymmetric designs in neural image compression.

2.5 Discriminators (GAN training)

Two discriminator families, trained jointly with the generator:

1. Wave-based (MelGAN-style multi-scale): the same convolutional discriminator applied to the waveform at original, 2×-, and 4×-downsampled resolutions. Each scale: initial conv → 4 grouped convs (group size 4, stride 4, channel multiplier 4, capped at 1024 channels) → 2 plain convs → logits.

2. STFT-based: complex STFT as real+imaginary channels, window W=1024, hop H=256. 2D conv (7×7, 32 ch) → 6 residual blocks alternating stride (1,2)/(2,2) along (time, frequency); final 1×(F/2⁶) conv aggregates frequency bins into per-time-step logits.

Both are fully convolutional → per-time-step logits, giving dense adversarial signal.

2.6 Losses (the GAN recipe from HiFi-GAN/MelGAN lineage)

With K+1 discriminators (k=0 STFT; k=1..3 wave scales), hinge losses:

• Discriminator loss (Eq. 1): L_D = E[ (1/K) Σ_k (1/T_k) Σ_t max(0, 1 − D_k,t(x)) + (1/K) Σ_k (1/T_k) Σ_t max(0, 1 + D_k,t(G(x))) ] — push logits of real audio above 1, generated below −1.
• Generator adversarial loss (Eq. 2): L_adv = E[ (1/K) Σ_k (1/T_k) Σ_t max(0, 1 − D_k,t(G(x))) ].
• Feature matching loss (Eq. 3): mean absolute difference of internal discriminator activations between real and generated audio — L_feat = E[ (1/KL) Σ_{k,l} (1/T_{k,l}) Σ_t |D^(l)_{k,t}(x) − D^(l)_{k,t}(G(x))| ]. Stabilizes GAN training and gives a perceptual mid-level reconstruction signal.
• Multi-scale spectral reconstruction loss (Eq. 4–5): mel-spectrograms S_s(x) with 64 mel bins, window s ∈ {2⁶,…,2¹¹} (64→2048), hop s/4; loss combines L1 plus a log-domain term with weight α_s = s/2:

L_rec = Σ_s Σ_t [ |S_s,t(x) − S_s,t(x̂)|₁ + α_s · ||log S_s,t(x) − log S_s,t(x̂)||₂ ].

• Total generator loss (Eq. 6): L_G = λ_adv·L_adv + λ_feat·L_feat + λ_rec·L_rec with λ_adv = 1, λ_feat = 100, λ_rec = 1.

The perception–distortion trade-off framing: reconstruction losses buy fidelity; the adversarial loss buys perceptual quality (plausible fine detail the L1 losses would blur).

2.7 Joint compression + enhancement (denoising)

Training data comes as (input, target, denoise) triples: if denoise=false, target = input; if true, target = clean speech. A binary, potentially time-varying conditioning signal is injected via FiLM layers (feature-wise linear modulation): a'_{n,c} = γ_{n,c}·a_{n,c} + β_{n,c}, with γ, β produced by a linear layer from the one-hot mode. FiLM is applied at the bottleneck (encoder- or decoder-side). This enables denoising toggled at inference with zero extra latency; denoising before quantization (encoder-side) also lowers the bitrate needed (~7–20% headroom vs. entropy coding was measured overall).

3. Training pipeline

• Data (all 24 kHz): clean speech — LibriTTS; noisy speech — LibriTTS mixed with Freesound noise (3 s crops, peak-normalized, noise gain uniform in [−30, 0] dB); music — MagnaTagATune; plus an internal real-world set of near/far-field reverberant speech. Eval: 200 clips (2–4 s), 50 per content type.
• End-to-end joint training of encoder, RVQ, decoder, and discriminators with the loss mix above; codebooks via EMA + k-means init + dead-code replacement; quantizer dropout for scalability.
• Model selection metric: ViSQOL (open-source, POLQA-correlated) — chosen because PESQ/POLQA are license-encumbered.
• Subjective eval: MUSHRA-inspired crowdsourced protocol, hidden reference, no anchor, 20 ratings per clip, native English speakers with headphones, plus rater screening.
• Baselines: Opus (IETF RFC 6716), EVS (3GPP), Lyra (Google's earlier 3 kbps generative codec).
• Deployment target: real time on a single CPU thread of a Pixel 4 smartphone — default model (32/32 channels) achieves RTF > 2.3× (encode 2.4×, decode 2.3×).

4. Key results and numbers

5. Why this matters for speech-capable language models

SoundStream is the progenitor of the audio token vocabularies that speech-language models are built on. The connection chain is direct:

1. Discrete audio tokens. RVQ turns any waveform into a sequence of discrete symbols (N_q codebook indices per 75 Hz frame). LLMs are sequence predictors over discrete tokens; RVQ is the bridge that makes speech a "language" an LLM can model. AudioLM, VALL-E, MusicLM, MusicGen, and Spirit-LM all use RVQ codecs directly descended from SoundStream (its successor EnCodec, and Google's own SoundStream variants) as their tokenizer; VALL-E famously uses EnCodec tokens with a transformer decoder to do zero-shot TTS.

2. The residual structure is a design pattern for LMs. Coarse quantizers capture prosody/speaker/timbre; fine quantizers capture acoustic detail. Hierarchical speech LMs (AudioLM's semantic→coarse→fine stages) exploit exactly this ordering: model the first few codebooks with one transformer, the rest with another.

3. Quantizer dropout → bitrate-scalable tokens prefigures variable-token-budget audio generation: the same tokenizer serves cheap (low n_q) and high-fidelity (high n_q) generation without retraining.

4. Engineering constraints that shaped the field: causal convolutions for streaming, real-time smartphone CPU decode, a single model across rates and content types — these defined the practical envelope that later textless-NLP and spoken-dialogue systems inherited.

5. GAN-decoder recipe. The adversarial + feature-matching + multi-scale mel reconstruction loss mix (with λ_feat = 100) became the standard recipe for neural vocoders and codecs (HiFi-GAN lineage → EnCodec → DAC), which is what makes decoded speech from tokens sound natural rather than buzzy.

In short: before an LLM can speak, something must convert speech into tokens an LLM can predict, and something must convert predicted tokens back into audio. SoundStream (2021) is the paper that established both halves with one clean, end-to-end-trainable, real-time architecture.

6. Caveats / limitations

• Reported bitrates are uncompressed codebook indices; entropy coding could save 7–20% but wasn't deployed.
• Constant bitrate per frame (no VBR); no explicit semantic content modeling — tokens are acoustic, not linguistic (later work added semantic tokenizers).
• Trained/evaluated at 24 kHz; music coverage limited to MagnaTagATune-style material.
• The "6 citations" figure in the extracted metadata reflects the preprint era; the paper is in reality one of the most-cited foundations of neural audio codecs.

Chapter 2 — AudioLM: a GPT for speech

Paper Digest: AudioLM — a Language Modeling Approach to Audio Generation

arXiv: 2209.03143 (September 2022, Google Research)

Authors: Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, Neil Zeghidour

Venue: Preprint (later appeared at IEEE TASLP-adjacent venues; foundational for VALL-E, MusicLM/SoundStorm, VALL-E 2, and the whole "audio-as-tokens" line)

---

1. The core problem

Raw audio is sampled at 16,000–44,100 values per second. If you naively treat audio like text and train an autoregressive Transformer to predict one sample at a time, you hit two walls:

1. Sequence length. Self-attention is O(n²); one second of 16 kHz audio is 16,000 tokens. Even 10 seconds is beyond practical context.

2. Structure. Models trained purely on waveforms (e.g., WaveNet) generate locally plausible but globally meaningless audio — "babbling speech" — because they never see abstractions like words, sentences, speaker identity, or musical phrase structure.

The insight of AudioLM is a two-level answer:

• Compress audio into discrete tokens at a manageable frame rate (tens of tokens/sec instead of tens of thousands of samples/sec), making audio generation literally a language-modeling problem: predict next token given previous tokens.
• But there is no single token type that does everything. You need two complementary token types, because:
• Tokens that reconstruct audio perfectly (neural codec tokens) lose the high-level structure — a LM on them produces speech with a consistent voice but nonsense linguistic content (babbling).
• Tokens that capture high-level structure (self-supervised "semantic" tokens) are not invertible — you cannot reconstruct a waveform from them.
• Hence the central contribution: a hybrid, hierarchical tokenization + multi-stage LM that models coarse-semantic tokens first, then conditions fine acoustic detail on them.

This paper is the "GPT moment" for speech: it shows a decoder-only Transformer, trained only on raw audio with zero transcripts, zero labels, zero phonetic annotations, learns syntax, semantics, speaker consistency, and studio-quality audio.

---

2. Background concepts (for a reader who knows transformers but not speech)

• Neural audio codec (SoundStream): a conv encoder compresses the waveform to embeddings at 50 Hz, a residual vector quantizer (RVQ) discretizes each embedding, and a conv decoder reconstructs the waveform. Trained end-to-end with reconstruction + adversarial (GAN) losses — this adversarial training is why quality is far above classic codecs like Opus at low bitrates.
• Residual vector quantization (RVQ): instead of one big codebook per frame, use a stack of Q codebooks. The first quantizer codes the embedding; the second codes the quantization error of the first; the third codes the error of the first two; and so on. Each codebook has N entries. Tokens from coarse layers carry the "big picture" (speaker identity, timbre); tokens from fine layers carry residual detail. This hierarchy is exactly what AudioLM exploits for its stage 2 / stage 3 split.
• Self-supervised speech model (w2v-BERT): a 0.6B-parameter Conformer (conv-augmented Transformer) pre-trained on speech with two objectives: contrastive learning and masked language modeling (MLM) — BERT-style span prediction over discretized speech units. Like BERT, it produces rich intermediate representations; like BERT, these are not designed to reconstruct the input.
• Semantic tokens: take an intermediate layer of the pre-trained w2v-BERT MLM module, normalize each dimension to zero mean/unit variance, run k-means with K clusters, and use the cluster indices as tokens. This is the same recipe as HuBERT-unit extraction in "textless NLP."
• ABX error rate: phonetic discriminability metric. Take minimal-pair trigrams ("bit" vs "bet"); measure how often a random instance X of one trigram is closer to an instance B of the other trigram than to another instance A of the same trigram. Lower = better phoneme separability.
• ViSQOL: a computational proxy for perceived audio similarity (like PESQ); higher (1–5 scale) = better reconstruction quality.
• sWUGGY / sBLIMP: zero-resource speech benchmarks. sWUGGY: does the model assign higher likelihood to a real word than a near-homophone non-word ("brick" vs "blick")? sBLIMP: does it prefer the grammatical of two near-identical sentences ("the dogs sleep" vs "the dog sleep")? Probes lexical and syntactic knowledge without any text.

---

3. Architecture: the full pipeline

Three frozen components wrap three trained Transformer LMs.

3.1 Tokenizer A — acoustic tokens (SoundStream)

• Convolutional encoder with 4 conv blocks, strides (2, 4, 5, 8) → product 320 → embeddings at 50 Hz from 16 kHz audio (a 320× temporal reduction).
• RVQ with Q = 12 quantizer layers, codebook size N = 1024 each → 600 tokens/sec at 6000 bps (50 × 12 × 10 bits).
• A coarser variant: Q = 4 layers → 200 tokens/sec at 2000 bps.
• Convolutional decoder maps the discrete codes back to the waveform. Trained end-to-end with reconstruction + adversarial losses.

3.2 Tokenizer B — semantic tokens (w2v-BERT + k-means)

• w2v-BERT XL: 0.6B parameters, Conformer-based, outputs 1024-dim features at 25 Hz (one vector per 40 ms; a 640× reduction from 16 kHz).
• Take the 7th layer of the MLM module (chosen by ABX / sWUGGY / sBLIMP sweeps + listening tests), per-dimension normalize, k-means with K = 1024 clusters → 25 semantic tokens/sec at 250 bps.
• Key property: these tokens encode what is being said (phonetics → words → syntax → semantics) but almost nothing about who says it or the recording conditions (see §4, Table III: speaker-classifier accuracy drops to 3.2% when only semantic tokens are fixed).

3.3 Why both? The trade-off table (Table I, LibriSpeech dev-clean)

Even at 6000 bps semantic tokens cannot reconstruct audio (ViSQOL 1.4); even at 6000 bps acoustic tokens have terrible phonetic discriminability. A LM trained only on acoustic tokens preserves voice and recording conditions from a prompt but produces babbling — this negative result motivates the whole design.

3.4 The three-stage hierarchical LM

Each stage is a separate decoder-only Transformer trained to predict next tokens; stages run sequentially at inference. RVQ's coarse-to-fine structure motivates the split.

• Stage 1 — Semantic modeling. Autoregressive LM over semantic tokens: p(z_t | z_<t). Captures long-term structure (linguistic content in speech; melody/harmony/rhythm in music).
• Stage 2 — Coarse acoustic modeling. Conditions on the full semantic sequence (prompt + generated), and predicts the flattened tokens of the first Q′ = 4 RVQ layers (200 tokens/sec → 800 tokens/sec after flattening): p(y_t^q | z, y^≤Q′_<t, y^<q_t). Coarse layers carry speaker identity and recording conditions. Flattening is row-major over time × quantizer index, with per-layer token offsets so each RVQ layer gets a distinct embedding table.
• Stage 3 — Fine acoustic modeling. Predicts the remaining 8 fine RVQ layers, conditioned only on the coarse acoustic tokens (semantic tokens are dropped — conditionally independent given coarse tokens). Runs on non-overlapping 3-second chunks, so it scales independently of sequence length and allows using more RVQ layers (Q = 12) for quality. Its job is removing the compression artifacts left by stage 2 (bitrate 2000 → 6000 bps).

Why three stages rather than one flat model?

1. Conditional-independence factorization: p(z_t | z_<t, y_<t) ≈ p(z_t | z_<t) — semantics don't depend on fine acoustics.

2. Shorter sequences per stage → much cheaper training/inference than one model over interleaved semantic+acoustic tokens.

3. Stage 3's chunking decouples its cost from total audio length.

Ratios: each semantic token (25 Hz) corresponds to 2 coarse acoustic frames (50 Hz), each with Q′ = 4 tokens → 2Q′ = 8 tokens per semantic token in stage 2, and 2(Q − Q′) = 16 tokens in stage 3.

3.5 Inference modes

• Unconditional generation: sample all semantic tokens from scratch, then stages 2–3. Produces diverse, syntactically/semantically consistent speech with varied speakers.
• Acoustic generation: use ground-truth semantic tokens from a real utterance, resample everything else. Content matches the original transcript (WER 6.0) while speaker/acoustics vary freely.
• Continuation (the headline use case): prompt with 3 seconds of unseen-speaker audio → extract its semantic + coarse acoustic tokens → stage 1 continues semantics, stage 2 continues coarse acoustics conditioned on the full semantic sequence plus prompt coarse tokens, stage 3 adds fine detail, SoundStream decodes the waveform. Continuations preserve voice, prosody, reverberation, background noise.

---

4. Training setup (concrete numbers)

• Dataset (speech): every component — SoundStream, w2v-BERT, k-means, and all three stage LMs — trained on Libri-Light unlab-60k: 60k hours of diverse, noisy English audiobook speech. No transcripts anywhere. Notably more robust than prior work (GSLM) which used the 6k-hour clean subset.
• Dataset (piano): internal dataset of 40k hours of piano recordings (beginner→expert, varied acoustic conditions). Evaluation prompts from MAESTRO.
• LM per stage (identical ×3): decoder-only Transformer, 12 layers, 16 heads, d_model 1024, FFN 4096, dropout 0.1, T5-style relative position embeddings → ~0.3B parameters per stage (~0.9B total).
• Optimization: random crops of 30 s / 10 s / 3 s equivalent input for stages 1/2/3; 16 TPUv4 per stage, batch size 256, 1M steps.
• Detail: consecutive repeated semantic tokens are removed during training in stages 1–2 (a practice from GSLM).
• Inference: temperature sampling — 0.6 / 0.8 / 0.6 for stages 1/2/3 (tuned trade-off between diversity and semantic consistency).
• Piano variant: SoundStream with Q = 3 RVQ layers, codebook size 2¹⁴ per layer (16384); stage 3 dropped — stage 2 predicts all 3 levels directly.

---

5. Key results

---

6. Why this matters for speech generation with LLMs

1. It established the recipe every subsequent system uses. Audio → discrete tokens → next-token prediction → decode. VALL-E (Microsoft, Jan 2023) is essentially AudioLM's acoustic side with text conditioning; MusicLM and SoundStorm (Google, 2023) build directly on the AudioLM token hierarchy; VALL-E 2, SPEAR-TTS, and AudioPaLM all inherit the semantic/coarse/fine cascade.

2. Proved the "LLM of speech" is learnable without text. sWUGGY/sBLIMP scores near or above supervised toplines show that next-token prediction on semantic tokens acquires lexicon and grammar — the strongest pre-LLM-era evidence that speech can be a first-class language-modeling modality.

3. The semantic/acoustic split is a general design pattern. Disentangling what is said (low-bitrate, structure-bearing tokens) from how it sounds (high-bitrate, detail-bearing tokens) is exactly how later systems separate content from speaker/style, enable zero-shot voice cloning from 3-second prompts, and get controllability (swap semantics, keep voice).

4. RVQ as a coarse-to-fine hierarchy. The Q′ coarse/fine split and the flattening-with-offsets trick became the standard handling of RVQ tokens in VALL-E, SoundStorm (which later parallelized this with masked modeling), and MusicGen.

5. Showed the practical necessity of cascade modeling. One monolithic LM over all tokens is intractable at audio scale; staged conditioning (semantic → coarse → fine, with the finest stage chunked) is the compute-feasible factorization — a template echoed in later audio, image, and video token cascades.

6. Set the responsible-AI template for voice cloning. 51.2% human indistinguishability + 98.6% machine detectability framed the deepfake-discussion that still surrounds speech LMs.

Limitations (as the paper itself notes)

• Continuation only — no TTS/text conditioning yet (explicitly flagged as future work; realized by SPEAR-TTS/VALL-E).
• English speech, single-instrument piano; semantic tokens from k-means are a heuristic layer choice (7th MLM layer, K=1024).
• Sampling is slow (three sequential AR stages; later fixed by SoundStorm's parallel decoding).

---

Source: Borsos et al., "AudioLM: a Language Modeling Approach to Audio Generation," arXiv:2209.03143 (2022). Full text extracted from the PDF; all numbers above are from the paper's Tables I–IV and Section IV.

Chapter 3 — Making Gemma 4 speak: the grafted speech head

Gemma 4 E4B Architecture — Digest & Speech-Head Analysis

Source: "Grafting a Speech Head onto Gemma 4 E4B" — Frisson Labs Blog.

Provenance per the article: Google's Gemma 4 E4B model card, the official E4B config, Google's audio-understanding docs, the authors' local MLX-VLM implementation inspection, and Kyutai's Moshi/Mimi paper.

1. Overall design

Gemma 4 E4B is a small, fast, multimodal text-decoder model shaped like a typical vision/audio-language model rather than a native speech-to-speech model:

• Inputs: text, image, audio (video handled as frame-based understanding — frames through the vision tower, not a separate video stream).
• Output: text tokens only.
• Core mental model: image and audio do not become separate output streams. They are converted into the same hidden-space language as text, inserted into the prompt at special placeholder-token positions, and a text decoder predicts the next text token.

Key numbers (spec panel)

• "E4B" ≈ effective 4B: the effective parameter count is 4.5B, so "e4b" refers to the effective ~4B-class size (the article gives 4.5B as the effective-params figure; it does not describe a separate total/sparse parameter count or MoE structure — no MoE is mentioned).

2. Attention & decoder

• 42 transformer decoder layers, width 2560.
• Hybrid attention: alternating sliding-window and global attention layers process the unified prompt (same hybrid pattern used in recent Gemma releases).
• Output head: tied text-embedding head producing 262K-vocab logits (weight tying between input embeddings and output head).

3. Input paths (per-modality)

• Text: 262K-vocab token IDs → 2560-dim embeddings; already in the decoder's hidden-space format.
• Image / video frames: vision tower → projector; projected features replace image placeholder slots in the sequence.
• Audio: audio tower → projector; projected features replace audio placeholder slots.

All modalities land in one 2560-wide shared sequence consumed by the 42-layer decoder.

4. Audio input handling

Concrete numbers from the article (attributed to Google's audio-understanding docs):

• Mono, 16 kHz float input.
• 32 ms frames at the audio-tower front end.
• Budget: 25 audio tokens per second of audio.
• Max clip length: 30 seconds.
• Training coverage of the audio tower is around speech recognition (ASR) and speech-to-text translation — it can listen and write text, nothing more.

5. Where Gemma stops (no native speech output)

• The audio tower is an input encoder only. Gemma 4 E4B:
• is not trained to produce Mimi-style discrete audio tokens,
• has no codec decoder to turn audio tokens into a waveform.
• Contrast with Moshi/Mimi (Kyutai): there, speech lives as discrete token streams and the model can generate speech the way an LM generates text. Gemma 4 E4B is "a multimodal text model that can hear," not speech-to-speech.

6. The Frisson experiment: grafting a speech head

Architecture

Gemma inputs (text / audio embeddings in one 2560-wide stream)
   → Frozen Gemma 4 E4B (42 layers)
   → TAP POINT: learned mix of the LAST 6 decoder layers
       (after the transformer stack, before the tied text-output head)
   → Trainable Gemma→Mimi audio head (only trained module)
   → Mimi codec tokens (8 codebooks @ 12.5 Hz)
   → Frozen Mimi decoder
   → 24 kHz waveform

• Tap point: a learned weighted mix of the last 6 of the 42 decoder layers, read before the tied vocabulary head — i.e., the speech branch is not downstream of decoded text (this is what distinguishes it from bolt-on TTS).
• Frozen: Gemma entirely; Mimi decoder entirely.
• Trainable: only the Gemma→Mimi token head — 152M trainable parameters.
• Loss: cross-entropy over Mimi codec-token IDs (8 codebooks), with teacher speech Mimi-encoded from WAVs. The head is a fixed parallel head (all 8 codebooks predicted in parallel; no temporal/AR speech modeling).

Inference discipline

• No Kyutai text prompt to Mimi — Mimi receives only predicted codec tokens.
• Text-driven samples: prompt template Say this naturally as speech:\n{text} fed to Gemma; no TTS in the loop.
• Audio-in → audio-out: a stricter run where the instruction+sentence lives entirely in the input WAV (mono audio object, zero text passed to Gemma). Gemma's audio-conditioned hidden states drive the same head.

Training setup

Results (smoke test — overfit architecture proof, not polished TTS)

• 7 text-driven clips from the step-500 checkpoint; Whisper found ≥2 target words in 5 of 7 samples.
• Best: input "the small adapter conditions clear speech through frozen layers" → Whisper heard "The small adapter conditions clear the picture through frozen windows" (6/7-ish word match). Worst cases garbled ("the voice sample conditions the final sample for the demo" → "of always simple conditions on and off", 1 word).
• Audio-driven clip (step-850): same "small adapter" sentence spoken in the WAV → Whisper heard "The small adapter conditions clear speeds through frozen mirrors" (6 matched words), proving the branch can be driven via the audio-input path.

What it tests vs. what's missing

• Tests: the tap point (late-layer hidden states can be trained to emit speech tokens) and the speech-token wiring (Gemma-owned learned branch + frozen Mimi as pure codec decoder).
• Not yet: speaking generated answers (requires running Gemma's normal AR decode loop and training the head on hidden states of generated tokens), generalization beyond memorized phrases, and quality (needs temporal modeling, duration control, streaming, stronger audio-token decoding, larger data).

7. Practical takeaway (Discord/game-buddy framing)

• Production-simple path: Gemma (text out) + low-latency external TTS voice layer — Gemma decides what to say; voice layer handles sound/interruption.
• Research path: the grafted audio head could eventually reduce service handoffs and keep multimodal nuance available to the voice — but it is a prototype, not a released Gemma capability.

Caveats

• The schematic chips in the article show flow/dimensionality, not real activations.
• All Gemma→Mimi samples are overfit smoke tests on tiny data; Whisper transcripts are noisy ASR of rough audio.
• The article does not give Gemma 4 E4B's total (non-effective) parameter count or any MoE/sparse-expert details; only "effective params 4.5B."

Chapter 4 — The full picture: build your own speaking model

Putting the three sources together, here is the canonical recipe for making a text LLM speak, distilled from 2021→now:

Step 1 — Get an audio tokenizer (SoundStream lineage)

Train (or take pretrained — EnCodec, Mimi, DAC) a codec: convolutional encoder → RVQ → GAN decoder. Decide your token budget: AudioLM used Q=12 codebooks × 1024 entries @ 50 Hz (600 tok/s); Mimi uses 8 codebooks @ 12.5 Hz. The RVQ hierarchy gives you a free coarse-to-fine ordering.

Step 2 — Get semantic tokens for structure

Audio without linguistic structure babbles. Extract tokens from a self-supervised speech model's intermediate layer (w2v-BERT layer 7, k-means K=1024 @ 25 Hz in AudioLM; HuBERT units in textless NLP). These carry what is said; acoustic tokens carry how it sounds.

Step 3 — Train hierarchical LMs

Stage A: AR LM over semantic tokens (the "what"). Stage B: AR LM over coarse acoustic RVQ layers, conditioned on semantics (the "who/style"). Stage C: fine layers, conditioned on coarse ones (the "polish") — chunked to keep cost tractable. Modern variations: text conditioning on top of semantics (VALL-E, SPEAR-TTS), or parallel masked prediction instead of AR for the acoustic stage (SoundStorm).

Step 4 — Wire it into your LLM (the Gemma 4 way)

If your LLM already hears (audio tower → placeholder tokens in the hidden stream), you don't need to retrain it: freeze everything, tap late decoder layers (last 6 of 42 in the Frisson experiment), and train a small head (152M params) to emit codec tokens; decode with the frozen codec decoder. Loss: cross-entropy on codec-token IDs with teacher-encoded speech.

What's still open (2026 frontier)

• Generating while speaking: the Frisson head only re-speaks prompted content; true speech-out requires training the head on hidden states of generated tokens inside the AR loop.
• Duration & streaming: fixed parallel token heads have no natural duration control; AR is slow; masked/parallel decoders (SoundStorm, Mimi-style) trade these off.
• Duplex conversation: Moshi-style full-duplex models model both speakers' streams simultaneously.
• Detection: humans were fooled (51.2% ≈ chance) but a simple classifier detected AudioLM speech at 98.6% — provenance and watermarking of synthetic speech remain unsolved.