The Voice as a Signature

The same sentence, spoken three times: cheerful, tired, angry. The words stay the same — but pitch, loudness and tempo reveal what the words leave unsaid. You make that measurable.

What it's about

Say the sentence "I'm fine" in three ways: cheerful, tired, angry. The words stay the same — and yet everyone immediately understands three different things. What carries the difference is not the words but the sound: the pitch, the tempo, the little breaks and pauses. In this sound lies a second message, often more honest than the first.

And the sound becomes readable the same way as everything in this book: you turn it into an image. A spectrogram shows the frequencies of the voice over time — and at once the same image recognition applies that otherwise tells cats from dogs. In this activity, though, we deliberately stay with the transparent features: pitch, loudness, tempo. You choose the features yourself — the light, explainable end of the toolbox.

A little background

What we measure. Three simple, telling quantities: the base pitch (how high the voice sits on average) and how much it varies (monotone vs. melodic); the loudness and its variation (energetic vs. flat); and the speaking tempo, roughly as the share of sounding to silent sections. From these few numbers comes the emotional fingerprint of a sentence.

As unique as a fingerprint. The voice carries two things at once: what you say and who you are. Its fine peculiarities allow speakers to be recognised — the basis of your phone listening for your voice. Exactly this uniqueness is used by the bonus at the end.

Recording

1. Install the packages. pip install librosa numpy soundfile. (For the extension also pip install openai-whisper.)
2. Speak the same sentence three times. Choose a short, neutral sentence — say "I'm fine" or "Today is Monday". Record it three times: once cheerful, once tired, once angry. Same text, same distance from the microphone.
3. Save as WAV. happy.wav, tired.wav, angry.wav. Quiet room, no echo, close enough to the microphone.
4. Analyse. Run the script below over all three and compare the numbers.

Analysing with librosa

About twenty lines. The code pulls the four features from each recording and lines them up. Full code on GitHub.

import librosa
import numpy as np

def features(file):
    y, sr = librosa.load(file)                   # load audio
    duration = librosa.get_duration(y=y, sr=sr)

    # pitch (fundamental frequency) only where the voice really sounds
    f0, voiced, _ = librosa.pyin(
        y, fmin=70, fmax=400, sr=sr)
    pitch  = np.nanmean(f0)                       # mean pitch in Hz
    melody = np.nanstd(f0)                        # variation = melody

    # loudness (energy) over time
    energy = librosa.feature.rms(y=y)[0]
    loud   = float(np.mean(energy))

    # speaking tempo, roughly: share of sounding to all sections
    voiced_share = float(np.mean(voiced))

    return dict(duration=duration, pitch=pitch, melody=melody,
                loud=loud, voiced_share=voiced_share)

for name in ["happy", "tired", "angry"]:
    m = features(f"{name}.wav")
    print(f"{name:6s}  pitch {m['pitch']:5.0f} Hz   "
          f"melody {m['melody']:5.0f}   loud {m['loud']:.3f}   "
          f"voiced {m['voiced_share']:.2f}")

Worksheet

When it sticks

Extension — separating the what from the how

So far you have measured the sound. Whisper, a large pre-trained speech model, reads the content — it turns the recording into text. So you hold both messages separately in your hands:

import whisper

model = whisper.load_model("base")           # small, runs without a graphics card
result = model.transcribe("angry.wav", language="en")
print("Said (the WHAT):", result["text"])
# The HOW you already measured above - pitch, melody, loudness.

The point: in all three recordings Whisper delivers the same text, while your sound features show three different moods. What was said is the same; how it was said is not.

Bonus — "Open sesame": the voice key

As a class you build a door opener from voice. Everyone records the same sentence several times; a small model learns from the sound features who is speaking — not what is said. Speak the sentence, and the model guesses the name.

1. Collect. Each person records the same sentence 5 times (so e.g. 5 people × 5 = 25 recordings). Names as filenames.
2. Pull features. For each recording the same sound features as above (plus optionally the MFCC, which librosa provides as librosa.feature.mfcc — a compact sound fingerprint).
3. Learn. A simple classifier (e.g. sklearn's KNeighborsClassifier) learns features → name. Hold back some recordings for testing.
4. Test. Speak a new recording — does the model guess the speaker? And the honest test: can someone with a similar voice (or a voice copy!) fool the key?

Food for thought

• The voice often reveals more than the words — and this second message is usually more honest, because it is harder to steer. Yet the same technology that makes it readable can also fake it. That is the heart of Chapter 10: honesty is not a fixed property of a signal but depends on how hard it is to fake — and that threshold is shifting.
• A read sound feature is a state, not a cause: "pressed, loud voice" does not necessarily mean "angry at you". The same caution as with the face — expression is not feeling.
• Because the voice is as unique as a fingerprint, it is useful and delicate at once. Who may record, store, reproduce your voice? This question grows more pressing with every better voice copy.