Dog and Horse

One photo is enough: you let a model guess an animal's mood — and set your own judgement against it. The most interesting part is where the two disagree.

What it's about

A dog cannot tell you it is afraid. But its body says it loud and clear: the tucked tail, the crouched posture, the ears laid back. Exactly this we exploited with my team and determined a dog's mood from a single photo — with a hit rate of 60 to 70 per cent. That sounds modest, but, as studies show, it beats most humans. For horses we read mainly ears, eyes and mouth; there too the model lands at about two out of three.

These research models are trained at great effort. For you we take a different, equally honest route: a ready-made model that connects images and language, CLIP. You hold it a photo and a few descriptions — "an anxious dog", "a relaxed dog" — and it says which description fits best. You train nothing; you only ask and look critically at the answer.

A little background

A shared alphabet of feelings. With what do we even label an animal's feelings? The book draws on an order by the neuropsychologist Jaak Panksepp, who identified in all mammals a few basic feelings — seeking, play, care, fear, rage, grief. The beauty: it is the same alphabet for dog, horse and at heart for us too. For horses "pain" is added as an especially important eighth feeling.

Where the AI looks. The research models use exactly the places where we read too: in the horse the ears, eyes and mouth; in the dog the whole posture including the tail. When a model was once let to sort the animal images with no labels at all, it even formed groups that fit none of our boxes — perhaps a hint at finer animal feelings for which we simply lack the words.

Photograph and judge for yourself

1. Install the packages. pip install transformers torch pillow.
2. Collect photos. Five to eight images of the same animal in clearly different situations: attentive, relaxed, playing, perhaps anxious. Frontal, good light, face and posture visible.
3. You first. Before the program, note your own judgement for each image from the alphabet of feelings. This matters — otherwise you unconsciously follow the machine.
4. Then the model. Let CLIP guess and lay the two judgements side by side.

Asking the model

CLIP understands English best, so the descriptions are English — the display stays in your language. The full code is on GitHub.

from transformers import CLIPModel, CLIPProcessor
from PIL import Image
import torch

model     = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")

# Panksepp alphabet: label -> English description for CLIP
feelings = {
    "Joy / play":  "a happy, playful dog",
    "Affection":   "a relaxed, affectionate dog",
    "Seeking":     "an alert, curious dog exploring",
    "Fear":        "a frightened, anxious dog",
    "Rage":        "an angry, aggressive dog",
    "Rest":        "a calm, resting dog",
}
labels  = list(feelings)
prompts = list(feelings.values())

def guess(image_file):
    image = Image.open(image_file).convert("RGB")
    inputs = processor(text=prompts, images=image, return_tensors="pt", padding=True)
    with torch.no_grad():
        logits = model(**inputs).logits_per_image[0]
    probs = logits.softmax(dim=0)               # descriptions -> probabilities
    best = int(probs.argmax())
    return labels[best], float(probs[best])

for file in ["dog1.jpg", "dog2.jpg", "dog3.jpg"]:
    label, conf = guess(file)
    print(f"{file}:  model guesses {label}  ({conf:.0%} sure)")

For horses

Swap the animal name in the code and add the eighth feeling. For horses pain matters especially:

feelings = {
    "Attentive":  "an alert, attentive horse with ears forward",
    "Relaxed":    "a calm, relaxed horse",
    "Fear":       "a frightened horse with ears back and wide eyes",
    "Agitation":  "an agitated, stressed horse",
    "Pain":       "a horse in pain, with tense face and eyes",
}

Horses, by the way, read us too — often more finely than we read them. Keep that in mind when you think about the limits of the machine next.

Worksheet

When it sticks

Food for thought

• Pure inference is powerful: a ready-made model becomes a tool without you training a single line. But "being able to use it" does not mean "trust it blindly" — the critical question stays your job.
• The animal Babel fish is a tool, not an oracle. A wagging tail is not always joy, and a model that learned on one breed reads another worse.
• The purpose decides: read for the animal's good — to notice pain or chronic stress early — such a reader is care. Used for exploitation, it is the opposite.

Extension — train your own (going further)

So far you only asked the model. Now you train one yourself — on real data, as in research. On Kaggle lies the dataset Pet's Facial Expression (kaggle.com/datasets/anshtanwar/pets-facial-expression-dataset) with faces of dogs, horses and cats, labelled happy, sad and angry. Instead of a heavy image network you use a light trick: CLIP gives each image a number vector — its "view" of the image — and on that, just like the cat in 12.1, a simple KNN learns.

import glob, os, numpy as np, torch
from PIL import Image
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import train_test_split

def image_vector(file):
    image = Image.open(file).convert("RGB")
    inputs = processor(images=image, return_tensors="pt")
    with torch.no_grad():
        return model.get_image_features(**inputs)[0].numpy()   # CLIP's view of the image

# folder per feeling (happy/ sad/ angry/) - folder name = label
FOLDER = "pets-facial-expression"
X, y = [], []
for path in glob.glob(os.path.join(FOLDER, "**", "*.jpg"), recursive=True):
    feeling = os.path.basename(os.path.dirname(path))
    X.append(image_vector(path)); y.append(feeling)
X = np.array(X)
print("read:", len(X), "images")

Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.25,
                                      random_state=0, stratify=y)
own = KNeighborsClassifier(n_neighbors=5).fit(Xtr, ytr)
print("hits on unseen images:", round(own.score(Xte, yte), 2))

Compare: does your self-trained KNN beat CLIP's blind guessing from the main part? And how does the picture change if you take only dogs or only horses?

• Where does the model look? With a heat map (Grad-CAM) you can show which image regions carry the decision. Check: does it look, like the research models, at ears, eyes and mouth — or at the background?
• Human and animal at once. Photograph animal and human in one scene. Does the mood of one match that of the other? That is the first step towards reading the interplay — and the bridge to the plants in the next part.