The Expression Mirror

Your webcam becomes a reader: a program reads, once a second, which feelings it suspects in your face — and you experience how pixels turn into probabilities.

What it's about

Even an infant reads faces before it understands a word. The face is the body's most honest page: it often moves faster than we can steer it, and reveals in fractions of a second what we might have wanted to hide. Exactly this readability makes it the most obvious target when a machine is to learn to read honest signals.

The machine's path is the same as in the AI toolbox: a camera delivers an image, a model first looks for fixed points — corners of the mouth, eyebrows, eyelids — and computes probabilities for emotional expressions from their positions relative to each other. It was trained on hundreds of thousands of hand-labelled face photos: again a work of open datasets and communities. On this book's map of AI, the expression mirror sits in the middle — a snapshot model that decides anew for every single image, with no memory. That matters for the next activity (9.2): only your analysis, not the model, turns it into a course over time.

A little background

From pixels to percentages. The model gives you not a truth but a distribution: "72 % happiness, 15 % surprise, …". These numbers are the model's confidence, not a measurement of your inner life. A raised corner of the mouth is a smile — not necessarily happiness. People smile out of politeness, embarrassment, even pain.

HSEmotion. We use an open, lightweight library for facial emotions (hsemotion) that runs on a small, pre-trained network — fast enough for live images, with no graphics card at all. The face search is handled by OpenCV. Both were built and given away by an open community; twenty years ago the same demo would have needed a research lab.

Setting up

1. Install the Python packages. In a console: pip install hsemotion opencv-python. On first run HSEmotion downloads its small model automatically.
2. Allow the camera. The operating system asks for camera permission on the first run — confirm it. Close other programs that occupy the camera (video chat!).
3. Start the program. Save the script (below) and run it. A window with your camera image opens, with a frame around your face and the recognised emotion.
4. Quit. Press the q key in the window.

The program

About thirty lines — all the heavy work sits in the two libraries. The full, commented code is on GitHub.

import cv2
from hsemotion.facial_emotions import HSEmotionRecognizer

# load the model (recognises happiness, sadness, anger, surprise, fear, disgust, neutral)
fer = HSEmotionRecognizer(model_name="enet_b0_8_best_afew")

# OpenCV's face finder
detector = cv2.CascadeClassifier(
    cv2.data.haarcascades + "haarcascade_frontalface_default.xml")

cam = cv2.VideoCapture(0)             # 0 = built-in camera
print("Window active - press q to quit.")

while True:
    ok, frame = cam.read()
    if not ok:
        break

    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
    faces = detector.detectMultiScale(gray, 1.3, 5)

    for (x, y, w, h) in faces:
        face_crop = frame[y:y+h, x:x+w]             # just the face
        emotion, probs = fer.predict_emotions(face_crop, logits=False)
        cv2.rectangle(frame, (x, y), (x+w, y+h), (0, 180, 0), 2)
        cv2.putText(frame, emotion, (x, y-10),
                    cv2.FONT_HERSHEY_SIMPLEX, 0.9, (0, 180, 0), 2)

    cv2.imshow("Expression Mirror", frame)
    if cv2.waitKey(1) & 0xFF == ord("q"):
        break

cam.release()
cv2.destroyAllWindows()

Worksheet

When it sticks

Food for thought

• The expression mirror measures an expression, not a feeling. Whoever makes that translation too quickly overestimates the technology — the most common mistake in the whole field. A smile does not mean "happy", a furrowed brow does not mean "angry".
• Pointed at yourself, the camera is a tool of self-knowledge; pointed at others without their knowledge, it is surveillance. The tool is the same — the difference lies solely in consent. Remember: what you find out about others belongs to them, not to you.
• The same image recognition that reads your face here reads a dog's posture in Part III and a plant's spectrogram in Part IV. One sense, lent to all the others — that is the secret lead character of this book.

Extension

• Show the probabilities. Have fer.predict_emotions(..., logits=False) give you the full distribution and draw the three strongest emotions with their percentages into the image. That way you see the model's hesitation, not just its favourite.
• Mirror in pairs. Sit facing each other (each filming themselves) and try to show the same expression. Does the model recognise you both equally well? A first, honest look at the question of fairness.
• Bridge to 9.2. Attach a timestamp to every prediction and write it to a file — that is already halfway to "Mood over an hour".