Hidden Signals · Companion site ← All activities DE·EN
Chapter 14 · Psychohistory · Activity 14.1

The Mood of the Swarm

Millions of headlines, one feeling: you measure the mood of a whole crowd over eight years and make visible when the world hoped — and when it trembled.

Duration 90 min Difficulty medium Group alone or in pairs Prerequisite Kaggle account (free) S

In a nutshell

What: You take eight years of daily news headlines, give each a mood value — from negative to positive — and average them into a single curve: the crowd's mood over time. Where the curve dips, something was happening.

The core idea: So far you have read individuals — a face, a voice, a plant. Now you read the crowd. In the sum of millions of traces the mood of a whole society becomes measurable — the old science-fiction idea of psychohistory, made real.

You need: the dataset "Daily News for Stock Market Prediction" from Kaggle and Python with pandas, vaderSentiment and matplotlib.

What it's about

In 1951 Isaac Asimov invented in Foundation a science that did not exist: psychohistory. Its inventor can calculate the fate of a whole empire in advance — not that of a single person, but that of the mass. For a long time this was pure fiction. Yet today each of us leaves traces without pause — posts, searches, headlines — and in the sum millions of such traces make an instrument for the mood of a society.

Back in 2010 we measured the mood of millions of Twitter messages over six months — simply by counting positive and negative words. Exactly this you rebuild here, only with a ready, clever word counter and eight years of news. In the end stands a single line: the emotion curve of the world.

Before you start — the data

Download from Kaggle the dataset Daily News for Stock Market Prediction (kaggle.com/datasets/aaron7sun/stocknews). In it sits RedditNews.csv with two columns: Date and News — the 25 most-read world news items per day, from 2008 to 2016. (You will only need the file DJIA_table.csv with the stock prices in 14.2.)

A little background

How do you measure mood? We use VADER, a ready-made mood reader built specially for short social-media texts. It knows for thousands of words how positive or negative they are, watches for intensifiers ("very"), negations ("not good") and even capitalisation — and gives in the end a single number between −1 (very negative) and +1 (very positive). You train nothing; you use a ready tool.

From many voices to one. A single headline says little. But the average of many headlines per day smooths out the chance and leaves a shared mood — just as the noise of many single measurements yields a signal only in the mean. That is the core of psychohistory: the individual is unpredictable, the crowd is not.

Getting the data

  1. Install the packages. pip install pandas vaderSentiment matplotlib.
  2. Load the dataset. Download from Kaggle, unzip and put RedditNews.csv next to your script.
  3. Take a quick look. pd.read_csv("RedditNews.csv").head() — two columns, Date and News. Now you know what you are working with.

Computing the mood

Each headline gets a value, then we average per day. The full code is on GitHub.

import pandas as pd
from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer

news = pd.read_csv("RedditNews.csv")             # columns: Date, News
vader = SentimentIntensityAnalyzer()             # ready mood reader (English)

# each headline -> a value between -1 (negative) and +1 (positive)
news["mood"] = news["News"].astype(str).apply(
    lambda text: vader.polarity_scores(text)["compound"])

# the crowd's mood per day = mean of all headlines of the day
daily = news.groupby("Date")["mood"].mean()
daily.index = pd.to_datetime(daily.index)
daily = daily.sort_index()
print(len(daily), "days  |  average mood:", round(daily.mean(), 3))

Drawing the mood curve

The daily values jump about — a rolling 30-day mean makes the trend visible.

import matplotlib.pyplot as plt

smoothed = daily.rolling(30, center=True).mean()    # 30-day mean against the noise
plt.figure(figsize=(11, 4))
plt.plot(daily.index, daily, color="#cfc6b4", lw=0.6, label="daily")
plt.plot(smoothed.index, smoothed, color="#2F5D3A", lw=2, label="30-day mean")
plt.axhline(0, color="#999", lw=0.8)
plt.ylabel("mood  (-1 negative ... +1 positive)")
plt.title("The mood of the swarm over eight years")
plt.legend(); plt.tight_layout(); plt.show()

What you should see

A jittery daily line and above it a calmer green trend curve, mostly slightly in the negative — world news is rarely cheerful. What is interesting are the dips: find the lowest points and look up what happened then (financial crisis, disasters, conflicts). Often the dent matches a real event — the curve has a memory.

Worksheet

Reading the world's mood

  1. At which two or three low points does the smoothed curve lie deepest? Research which event might match it.
  2. Why do we average over many headlines and over 30 days? What do you gain, what do you lose?
  3. VADER reads words, not meaning. Name an example (say, irony or an ambiguous word) where such a tool must go wrong.
  4. The curve lies almost throughout in the negative. Does that mean the world was mostly bad — or does it say more about the news?
  5. "The individual you cannot predict, the crowd you can." Explain from your curve why averaging many single voices is what makes a readable mood in the first place.
Show solution

1. Individual. Often the curve falls around the 2008/09 financial crisis and in phases of escalating conflict. The aim is to link a dip with a real event — and to notice that the link is plausible, not compelling.

2. The average of many headlines removes the chance of single words; the 30-day mean removes the daily jitter and shows the trend. Gain: a calm, readable signal. Loss: short, real spikes (a single fierce day) vanish.

3. For example "not bad" (VADER usually catches the negation, but not irony like "great, another crisis") or a word like "positive" in "positive test result", meant negatively. A word counter has no context.

4. More the latter: news reports disproportionately on problems and conflict, not on the calm everyday. The curve measures the mood of the reporting, not directly that of the world — an important difference.

5. A single headline is almost chance — now positive, now negative. Only the mean of many lifts the shared signal out of the noise. That is exactly why the crowd is readable where the individual is not.

When it sticks

ProblemLikely cause & fix
KeyError: 'News'The column is named differently (in some versions Headlines). Check with df.columns and adjust the name.
Text starts with b'...'The headlines are stored as byte strings. Strip the leading b' and trailing ', e.g. with str.strip("b'\"").
Very slowScoring ~73,000 headlines takes a while. Compute once and cache the result with to_csv.
Curve is a straight lineDate was not sorted as a date. Do not forget pd.to_datetime and sort_index().
All values are 0VADER got empty or non-text values. Guard with .astype(str) and drop empty rows.

Food for thought

Extension

← 13.2 Plant Emotions with AI 14.2 Asimov's Psychohistory →