Chapter 14 · Psychohistory · Activity 14.2

Asimov's Psychohistory

Does the crowd's mood predict the stock market? You lay the mood curve against the Dow Jones — and learn from the honest result why Asimov gave his psychohistory a "Mule".

Duration double lesson Difficulty high Group alone or in pairs Prerequisite Activity 14.1 S

In a nutshell

What: You take the daily mood from 14.1 and lay it beside the daily return of the Dow Jones. Then you test two things: does yesterday's mood relate to today's price? And can a model guess from the mood alone whether the market rises or falls?

The core idea: This is the touchstone of psychohistory. Back in 2010 we tried to predict market moves from Twitter mood. Here you rebuild it — and meet the chapter's most honest lesson: the crowd is a little predictable, but never fully. Exactly there the Mule waits.

You need: the same Kaggle dataset as in 14.1 (now also DJIA_table.csv) and Python with pandas, vaderSentiment, scipy and scikit-learn.

What it's about

Asimov was too clever to make his psychohistory all-powerful. In the novels appears a figure he calls the Mule: a single, unforeseen person who overturns Seldon's whole calculation. The crowd follows patterns — until an outlier comes that no pattern knew.

Exactly this tension you now measure. The guess: when the crowd's mood tips, it ought to show in the markets — fear pushes prices down, confidence lifts them. You check that not by gut feeling but on eight years of data. And you will see: a spark of connection is there — but whoever wants to make sure money from it meets the Mule.

Before you start — the data

From the same Kaggle dataset (aaron7sun/stocknews) you now also need DJIA_table.csv — the daily prices of the Dow Jones index (columns Date, Open, High, Low, Close …). RedditNews.csv supplies the headlines as in 14.1.

A little background

Why "yesterday against today"? A prediction is only one if the cause lies before the effect. So we set yesterday's mood against today's return. If we found a connection there, it would mean: the mood runs ahead of the price — you could use it to predict. Comparing the same day with itself, by contrast, only measures that bad news and falling prices occur together — that is no prediction.

What an honest result is. Expect no miracles. If a model reached 90 per cent, that would be an alarm bell for an error (usually a data leak). A few points above the chance line are the realistic, honest result — and even that is remarkable, considering it comes from the tone of headlines alone.

Joining mood and market

First the mood as in 14.1, then the market's return, then joining the two by date. The full code is on GitHub.

import pandas as pd
from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer

# 1) daily mood (as in 14.1)
news = pd.read_csv("RedditNews.csv")
vader = SentimentIntensityAnalyzer()
news["mood"] = news["News"].astype(str).apply(
    lambda t: vader.polarity_scores(t)["compound"])
mood = news.groupby("Date")["mood"].mean().rename("mood")
mood.index = pd.to_datetime(mood.index)

# 2) the market: daily return of the Dow Jones
market = pd.read_csv("DJIA_table.csv", parse_dates=["Date"]).sort_values("Date")
market["return"] = market["Close"].pct_change()

# 3) join the two by date; shift the mood by one day
tab = market.merge(mood.reset_index(), on="Date")
tab["mood_yesterday"] = tab["mood"].shift(1)      # yesterday's mood
tab = tab.dropna(subset=["mood_yesterday", "return"])
print(len(tab), "shared trading days")

Does the mood predict the market?

Two checks: a simple connection and a small prediction model — both measured honestly against the chance line.

import numpy as np
from scipy.stats import pearsonr
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score

# a) connection: mood(yesterday) vs return(today)
r, p = pearsonr(tab["mood_yesterday"], tab["return"])
print(f"correlation mood(yesterday) -> return(today):  r = {r:.3f},  p = {p:.2f}")

# b) prediction: 'up or down' from yesterday's mood alone
X = tab[["mood_yesterday"]].values
y = (tab["return"] > 0).astype(int)               # 1 = market rises
score  = cross_val_score(RandomForestClassifier(200, random_state=0), X, y, cv=5).mean()
chance = max(np.bincount(y)) / len(y)
print(f"hits 'up/down':  {score:.2f}   (chance line: {chance:.2f})")

What you should see

The correlation is tiny and mostly barely significant; the model lands near the chance line, perhaps a few points above. That is not a failure — it is the honest answer: from the mere tone of the news the market can at best be sensed a little, not predicted for sure. Whoever expected a money machine here has just met the Mule.

Worksheet

The limits of prediction

How large is your correlation, how high your hit rate against the chance line? Would you invest your savings on it?
Why do we compare yesterday's mood with today's return — and not both from the same day? What would be the fallacy with the same day?
Suppose a model reached 95 per cent. Why would you be suspicious rather than delighted?
What is the "Mule" in this task? Name a real event that no mood curve could have foreseen.
Imagine the prediction got much better. Who could do good with it — and who harm? Name one example each.

Show solution

1. Individual — typically a very small correlation and a hit rate near the chance line. Honest answer: no, for an investment it is far too uncertain; the spread is huge against the tiny edge.

2. A prediction requires the cause to lie before the effect. On the same day you only measure that bad news and falling prices occur together — that is backwards and no use for predicting; you would already know the outcome.

3. Because 95 per cent on such a noisy problem almost certainly points to an error — usually a "data leak" where information about the future accidentally reaches the features. Too-good results are a warning, not a triumph.

4. The Mule is the unforeseeable: an assassination, a surprise central-bank decision, a natural disaster, a single tweet from a powerful person. No mood mean over millions can pre-empt such a single blow.

5. Good: early warning of panic, crisis aid, spotting collective distress. Harm: manipulation of elections or markets, surveillance, deliberately stoking moods. The same power, two directions — which is why responsibility belongs to it.

When it sticks

Problem	Likely cause & fix
`merge` yields 0 rows	Date formats do not match. Bring both sides to a real date with `pd.to_datetime` before joining.
Very high hits (> 0.9)	Almost certainly a data leak — e.g. today's return (or `Close`) accidentally sits in `X`. Use only `mood_yesterday` as a feature.
Hits vary a lot per run	Little signal, much noise. Average `cross_val_score` over several folds and place the chance line next to it.
Correlation is `nan`	Missing values from the shift. Drop the first row (with no "yesterday") using `dropna`.
Different column names	Some versions name the price column differently. Check with `market.columns` and adjust `Close`.

Food for thought

Psychohistory is real — but modest. The crowd follows patterns, yet the Mule, the unforeseen, remains. An honest "a little predictable" is worth more than a boastful "predictable".
Before we come to the brightest use of this power — understanding how groups create together — stands the warning: the same tools that measure moods can also steer them. Whoever can read the crowd is tempted to guide it.
That is exactly why responsibility is no accessory but the core: a tool that predicts the crowd is as good or as dangerous as the hand that wields it.

Extension

More lead time. Try the mood from two days ago or a 3-day mean as a feature. Does the prediction improve — or does the Mule remain?
Use the ready label. The dataset already holds in Combined_News_DJIA.csv a label (1 = market rises). Compare your result with a model trained directly on it.
A different event. Instead of the market: lay the mood curve against the data of an election or a crisis. Do you see a clearer lead there than with the prices?