Chapter 3: Linear Regression
Two Hundred Years of Regression
The story of linear regression begins with a 23-year-old mathematician in 1809. Carl Friedrich Gauss, working to predict the orbit of the asteroid Ceres, invented the method of least squares — the same computational backbone that powers every regression output you will run in this chapter. His insight was elegant: given noisy measurements, find the line that minimizes the sum of squared errors. He published it almost as an afterthought in a book about celestial mechanics, unaware that he had created the most widely used statistical technique in the history of science.
Seventy-six years later, the British scientist Francis Galton was studying the heights of parents and their adult children. He noticed something surprising: tall parents tend to have children who are tall, but not quite as tall; short parents tend to have children who are short, but not quite as short. He called this phenomenon “regression to mediocrity” — the tendency of extreme values to pull back toward the average across generations. The word “regression” stuck, even though modern regression does far more than describe hereditary reversion. Galton’s student Karl Pearson formalized the correlation coefficient and the mathematics of simple linear regression in 1896, and Ronald A. Fisher in the 1920s extended the framework to multiple predictors, analysis of variance, and the significance tests you will use throughout this chapter.
Why Regression Dominates Business Analytics
Regression is not merely a statistical technique — it is the language that business uses to quantify relationships. A McKinsey study of Fortune 500 analytics practices consistently finds that regression (linear and logistic) is the most frequently deployed model in production environments, ahead of decision trees, neural networks, and everything else. Why? Because regression produces interpretable coefficients: a single number that says “each additional dollar of median household income in a neighborhood adds $X to expected pharmacy profit, holding all other demographic features constant.” That ceteris paribus interpretation is what makes regression indispensable in strategy, marketing, operations, and finance.
The finance applications are particularly powerful, and carry Nobel Prize provenance. William Sharpe shared the 1990 Nobel Memorial Prize in Economic Sciences largely for the Capital Asset Pricing Model (CAPM) — a two-parameter regression that defined how Wall Street prices risk for the next three decades. CAPM’s slope coefficient, beta (\(\beta\)), became the lingua franca of equity analysis: aggressive stocks with \(\beta > 1\) amplify market swings; defensive stocks with \(\beta < 1\) dampen them. Every Bloomberg terminal displays beta. Every equity research report references it.
Eugene Fama shared the 2013 Nobel Prize (with Lars Peter Hansen and Robert Shiller) in large part for demonstrating that CAPM was incomplete. With Kenneth French, Fama documented that small-cap stocks and value stocks earned returns that the market factor alone could not explain. The resulting Fama-French factor models — first three factors, then five — replaced CAPM as the benchmark for risk adjustment in academic research and in institutional portfolio management. Today, factor investing based on Fama-French principles accounts for over $1 trillion in assets under management at firms like AQR Capital Management, Dimensional Fund Advisors (DFA), and BlackRock.
What You Will Learn
This chapter follows the natural workflow of applied regression analysis:
- Model — write down the mathematical relationship between variables
- Estimate — fit the model using OLS (Ordinary Least Squares)
- Infer — test hypotheses and build confidence intervals for coefficients
- Diagnose — check the LINE assumptions that make inference valid
- Predict — generate point forecasts and prediction intervals for new observations
- Select — choose the right set of predictors using AIC, BIC, and adjusted \(R^2\)
Each step connects to a real finance or business application. You will fit CAPM to NVDA stock returns, test whether NVDA has earned Jensen’s alpha, extend to the Fama-French five-factor model, and predict profits for a pharmacy chain using multiple regression. By the end, you will be able to read a regression output table the way a quant reads a Bloomberg screen — fluently, critically, and with precise understanding of what every number means.
Why This Chapter Is Weighted at 30%
Regression is the foundation on which Chapters 4 through 6 rest. Clustering (Chapter 4) uses distance metrics that assume the same geometric intuition as regression residuals. Classification models (Chapter 5) are direct extensions of the regression framework to binary outcomes. Time series forecasting (Chapter 6) adapts regression to handle serial dependence. If you understand regression deeply — not just the mechanics but the statistical logic — every subsequent topic becomes easier. The 30% weight reflects this foundational status. Master regression, and you have mastered the core reasoning pattern of quantitative business analysis.
Model → Estimate → Infer → Diagnose → Predict → Select.
Every section in this chapter maps to one step in this workflow. Keep the pipeline in mind as you progress.
Correlation & Visualisation
The Logic of Prediction
Prediction is what separates data-driven decisions from guesswork. A regression model takes the historical relationships embedded in a dataset and translates them into quantitative statements about the future: given that one variable has moved, by how much should the other be expected to move, and with what residual uncertainty. The discipline of regression rests on three questions a business analyst will return to repeatedly throughout this chapter. Is a particular stock — say NVDA — aggressive or defensive relative to the broader market? Which underlying economic forces drive its day-to-day returns? And once those forces are accounted for, is there any genuine excess return — what finance practitioners call alpha — that remains? Each of these questions translates into a regression specification, an estimated coefficient, and a hypothesis test. The Capital Asset Pricing Model that opens this chapter fits a line through NVDA returns plotted against market returns: the slope of that line is the stock’s market sensitivity, the intercept is its alpha, and the scatter around the line is the idiosyncratic risk that diversification can erase.
Visualising Bivariate Association
Background: The History of Correlation
The idea of measuring association between two variables has a surprisingly rich intellectual history. Francis Galton’s famous quincunx (a peg-board that demonstrated the normal distribution physically) was not just a parlor trick — it was Galton’s attempt to visualize how two generations of heights co-varied. In 1885, Galton drew scatter plots of parent heights against child heights and noticed the now-famous regression-to-the-mean effect. He labeled the slope of this scatter plot the “index of co-relation.”
It was his student Karl Pearson who cleaned up the mathematics in 1896, deriving the formula we use today. The Pearson correlation \(r\) has a beautiful theoretical justification: it is the cosine of the angle between the two mean-centered data vectors in \(n\)-dimensional space. That geometric interpretation is why \(r\) is bounded in \([-1, +1]\) — it follows directly from the Cauchy-Schwarz inequality, one of the most fundamental inequalities in mathematics:
\[|\langle \mathbf{u}, \mathbf{v} \rangle| \le \|\mathbf{u}\| \cdot \|\mathbf{v}\|\]
applied to the centered vectors \(\mathbf{u} = (x_1 - \bar{x}, \ldots, x_n - \bar{x})\) and \(\mathbf{v} = (y_1 - \bar{y}, \ldots, y_n - \bar{y})\).
The most important warning in statistics: correlation is not causation. Galton’s discovery that taller parents have taller children is a correlation. Whether height is caused by genetics, nutrition, or social environment requires a different type of evidence. The computer scientist and philosopher Judea Pearl (Turing Award, 2011) formalized this distinction through his do-calculus: the difference between observing that \(X = x\) and intervening to set \(X = x\) is the difference between correlation and causation. In finance, correlation between two stock returns does not mean one causes the other — they may both respond to a common factor (the market). This is precisely why CAPM uses regression (which quantifies a directional relationship) rather than correlation alone.
The classic cautionary tale: ice cream sales correlate with drowning rates. Both rise in summer. Neither causes the other. Always ask: is there a confounding variable?
Correlation Coefficient
The Pearson correlation \(r\) measures the strength of the linear association between two variables and is defined as the standardised covariance:
\[ r = \frac{\sum_{i=1}^{n}(x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum_{i=1}^{n}(x_i - \bar{x})^2 \cdot \sum_{i=1}^{n}(y_i - \bar{y})^2}} \]
The numerator is the sample covariance of \(x\) and \(y\), and the denominator normalises by the product of the two standard deviations. The result is bounded between \(-1\) and \(+1\). A value of \(+1\) indicates a perfect positive linear relationship: every observation lies exactly on an upward-sloping line. A value of \(-1\) indicates the same in the opposite direction. Intermediate values describe weaker association, with the conventional reading that \(|r|\) around \(0.5\) represents a strong linear pattern, \(|r|\) near \(0.2\) a modest pattern, and \(|r|\) near zero an absence of linear structure. The metric measures linear association only — two variables may be tightly coupled through a quadratic or trigonometric relationship and still yield \(r \approx 0\), which is why visual inspection of a scatter plot must always accompany the numerical correlation.
Loading the Data and Computing Correlation
The dataset accompanying this book contains real daily closing prices for NVDA and SPY over 2023 and 2024, packaged as a compact CSV that lives alongside the book itself. The analysis that follows runs entirely in the browser through Pyodide, with no external download or live API call required. This design choice matters for reproducibility: the same numbers appear in the same cells regardless of when the reader executes the code or what financial data provider they happen to subscribe to.
Before running, guess the off-diagonal entry: do you expect NVDA-versus-SPY daily-return correlation to land near 0.2, 0.5, or 0.8 — and what does each value imply about NVDA as a single-stock proxy for the market?
The browser fetched a small CSV (~25 KB) hosted alongside this book, parsed it into a pandas DataFrame, and computed daily returns + correlation — all in your browser.
Interpretation. The correlation matrix shows how tightly NVDA and SPY daily returns move together over 2023–2024. A typical value of \(r \approx 0.74\) measures the strength and direction of the linear relationship. To translate \(r\) into “share of variance explained,” square it: \(r^2 \approx 0.55\), meaning about 55% of the variation in NVDA’s daily returns can be linearly attributed to co-movement with the S&P 500, leaving roughly 45% as idiosyncratic — NVDA-specific news such as earnings surprises, GPU demand reports, and regulatory announcements. (A common mistake is to read \(r\) itself as a percentage of explained movement; that’s \(r^2\), not \(r\).) Notice that the diagonal is always 1 (a return is perfectly correlated with itself), and the off-diagonal \(r\) is symmetric: correlation between NVDA and SPY equals correlation between SPY and NVDA.
The key limitation: this single number compresses the entire relationship into a scalar. It tells you the strength of the linear association, but not the slope (how much NVDA moves per 1% SPY move — that requires regression), nor the direction of any causal mechanism.
Simulated NVDA vs SPY in Pyodide
Scatter Plot: NVDA vs. SPY
A slope greater than one is the geometric signature of an aggressive stock: NVDA moves more than the market on any given day, in either direction.
The scatter plot with fitted line makes the relationship visceral. Each dot is one trading day; its horizontal position is the SPY return and its vertical position is the NVDA return. The red regression line — slope approximately 1.9 in this simulation — slices through the cloud at a steep angle. On a day when SPY drops 2 per cent, the line predicts NVDA falls roughly 3.8 per cent. The scatter around the line is the idiosyncratic, firm-specific noise that the market cannot predict.
Notice that the cloud is elongated diagonally from lower-left to upper-right — this is positive correlation made visual. If the slope were exactly 1.0, the cloud would run along a 45-degree angle and NVDA would behave exactly like the market (a perfect index fund). The slope exceeding 1.0 is the geometric fingerprint of an aggressive stock — one that amplifies both market rallies and market selloffs.
Every equity analyst at Goldman Sachs, Morgan Stanley, and JPMorgan runs this exact chart when initiating coverage on a new stock. The slope (beta) goes directly into the valuation model. A stock with \(\beta = 2\) gets a higher required return (cost of equity) in the discounted cash flow model — investors demand compensation for taking on extra market risk.
Common pitfall: Don’t confuse correlation with slope. A high correlation (\(r\) near 1) does not mean the slope is near 1. Two variables can be perfectly correlated but have a slope of 0.1 or 10. Regression gives you the magnitude of the relationship; correlation only gives relative strength.
Simple Linear Regression (CAPM)
Simple Linear Regression
Background: Sharpe’s Revolution
In 1964, William Sharpe published “Capital Asset Prices: A Theory of Market Equilibrium under Conditions of Risk” in the Journal of Finance. It was a remarkable paper for its simplicity: with one slope coefficient, it claimed to explain why different stocks should earn different expected returns in equilibrium. The key insight was that only systematic risk — risk correlated with the market — commands a premium. Idiosyncratic risk (the scatter around the regression line) can be diversified away by holding a portfolio, so the market pays nothing for bearing it.
CAPM represented the first rigorous translation of risk into a quantifiable, tradeable concept. Before Sharpe, “risk” in finance was largely qualitative. After Sharpe, every stock had a number — its beta — and that number determined its cost of equity. The implication for corporate finance was profound: a project funded by a high-beta firm needs a higher hurdle rate than the same project funded by a low-beta utility.
The model attracted fierce criticism. Richard Roll’s critique (1977) pointed out that the “market portfolio” in CAPM theory is unobservable — it should include every investable asset (stocks, bonds, real estate, human capital), not just the S&P 500. Using SPY as a proxy introduces measurement error that makes any empirical test of CAPM inconclusive. Despite this, beta became (and remains) the default risk measure in equity research, corporate finance, and regulatory rate-setting for utilities worldwide. Sharpe shared the 1990 Nobel Prize with Harry Markowitz (portfolio theory) and Merton Miller (capital structure) — the three pillars of modern financial economics.
At Goldman Sachs equity research, every stock in coverage has a Bloomberg-supplied beta prominently displayed in the first row of each research note. Portfolio managers use betas to construct “market-neutral” portfolios (long low-beta, short high-beta) that profit from mispricing without taking on net market exposure. CAPM’s simplicity is a feature, not a bug — it communicates risk in a common language the entire financial industry speaks.
Simple Linear Regression Model
\[ y = \alpha + \beta x + \epsilon \]
Geometrically, the regression line passes through the cloud of observations. The intercept \(\alpha\) is the height of the line where it crosses the vertical axis at \(x = 0\), representing the predicted value of \(y\) in the absence of any contribution from \(x\). The slope \(\beta\) measures the change in the predicted value of \(y\) associated with a one-unit increase in \(x\), which appears visually as the tilt of the line. The error term \(\epsilon_i\) is the vertical distance from each individual observation to the line — the residual that the linear model cannot explain. Estimating this model amounts to choosing the values of \(\alpha\) and \(\beta\) that minimise the sum of squared vertical distances across all observations.
CAPM: A Canonical Simple Regression
The Capital Asset Pricing Model is a simple linear regression of a stock’s excess return on the market’s excess return:
\[ Y = \alpha + \beta\, X + \epsilon \]
Here \(Y = R_{\text{NVDA}} - R_f\) denotes the NVDA excess return and \(X = R_m - R_f\) denotes the market excess return, both measured net of the risk-free rate. The intercept \(\alpha\), known in finance as Jensen’s alpha, captures the average daily return that NVDA earns above what its market exposure predicts. A positive alpha means the stock has outperformed its risk-adjusted benchmark; a value indistinguishable from zero is consistent with an efficient market in which no security earns a free lunch. The slope \(\beta\), known as market beta, measures the sensitivity of the stock to broad market movements. A beta above one identifies an aggressive stock that amplifies market moves in both directions, while a beta below one identifies a defensive stock whose returns are dampened relative to the market.
Preparing the Data
We subtract \(R_f\) because CAPM measures the premium over T-bills.
Fitting the CAPM Model
Three steps: (1) sm.add_constant(X) adds \(\alpha\), (2) sm.OLS(y, X) defines the model, (3) .fit() estimates.
.fit() uses Ordinary Least Squares (OLS) — the same method from ISOM 2500. It finds \(\hat{\alpha}, \hat{\beta}\) that minimise \(\sum (y_i - \hat{y}_i)^2\).
The model.summary() output is dense, and it rewards a systematic reading. The estimated slope \(\hat{\beta} \approx 2.22\) implies that NVDA moves approximately 2.22 per cent for every one per cent move in the market. Translating this into capital exposure: a holding of $10{,}000 in NVDA confronted with a ten per cent market decline can expect to lose approximately $2{,}220 from the systematic component alone, before adding the idiosyncratic noise. This is the leverage embedded in any aggressive technology stock. The estimated intercept \(\hat{\alpha} \approx 0.0043\) corresponds to a daily alpha of about 43 basis points. Simply scaled to a year of 252 trading days, that is \(0.0043 \times 252 \approx 1.08\), i.e. roughly 108 per cent per year — an obviously implausible figure, because the simulation deliberately injected a non-zero alpha for illustration; no real stock could sustain alpha at this magnitude in an efficient market. In real return data, daily alphas are an order of magnitude smaller and rarely statistically distinguishable from zero. Whether this estimate is statistically significant determines whether the analyst can claim that NVDA genuinely outperforms its risk-adjusted benchmark. The coefficient of determination \(R^2 \approx 0.54\) tells the analyst that the market explains roughly fifty-four per cent of NVDA’s daily return variance, leaving the remaining forty-six per cent to firm-specific news — GPU shipment data, data centre contracts, analyst upgrades, and the like — which is precisely the diversifiable component that disappears in a sufficiently large portfolio. With \(n = 400\) training observations, representing eighty per cent of the five hundred trading days in the sample, the \(t\)-distribution is reliable and the Central Limit Theorem keeps the inference well-behaved even when the residuals depart from perfect normality.
A subtle but important caveat applies to the alpha test in this simulated dataset. Finding \(p(\alpha) < 0.05\) in a simulation whose data-generating process built in a positive alpha is mechanically expected — the test recovers what the simulation injected. The genuine empirical question is whether a real stock, with no known underlying alpha, exhibits a statistically significant intercept. In real return data, most stocks do not, and this is a direct implication of the efficient market hypothesis: a free lunch large enough to detect would attract capital until the apparent alpha collapsed.
Reading the Regression Output
The condensed regression output looks like this:
| coef | std err | t | P>|t| | |
|---|---|---|---|---|
const (\(\alpha\)) |
0.0043 | 0.001 | 4.32 | 0.000 |
Mkt_excess (\(\beta\)) |
2.221 | 0.103 | 21.65 | 0.000 |
| \(R^2\) | 0.542 |
Three numbers carry the story. The slope estimate \(\hat{\beta} \approx 2.22\) says that NVDA moves a little over two per cent for every one per cent move in the market. Its \(p\)-value of essentially zero (the column reports 0.000 because the value is smaller than the table’s display precision) leaves no room for doubt that the market is a significant predictor of NVDA’s daily returns. The intercept’s \(p\)-value is similarly tiny, so the null of zero alpha is rejected at any conventional significance level — though, as noted above, this is by construction in a simulation that injected a positive daily alpha. The \(R^2\) of approximately 0.54 indicates that the market explains roughly half of NVDA’s day-to-day return variance, leaving the remaining half as idiosyncratic risk that diversification can erase.
Decomposing the Variation
In any linear regression, the total variation in \(Y\) decomposes additively into a part explained by the regression and a part left unexplained:
\[ \underbrace{SST}_{\sum(y_i - \bar{y})^2} \;=\; \underbrace{SSR}_{\sum(\hat{y}_i - \bar{y})^2} \;+\; \underbrace{SSE}_{\sum(y_i - \hat{y}_i)^2} \qquad R^2 = \frac{SSR}{SST} \]
The total sum of squares, \(SST\), measures the spread of the observed \(y\) values around their mean. The regression sum of squares, \(SSR\), measures how much of that spread is captured by the fitted line — equivalently, the spread of the predictions \(\hat{y}_i\) around the mean. The error sum of squares, \(SSE\), measures the residual scatter of the observations around the fitted values. The coefficient of determination \(R^2\) is simply the share \(SSR/SST\), the proportion of total variation that the regression explains.
In the CAPM, this decomposition has a direct financial interpretation: \(SSR\) measures the systematic risk captured by the market factor, while \(SSE\) measures the idiosyncratic risk that is specific to the stock itself.
The variation decomposition is the balance sheet of a regression. The total variance in NVDA returns is partitioned by the regression into a share that the market explains and a share that the market cannot reach. The explained share, \(SSR\), is the part of the variance that loads on systematic risk — exposure that cannot be diversified away because every stock in the market is to some degree exposed to the same broad factor. The unexplained share, \(SSE\), is the idiosyncratic variance unique to NVDA, the portion that a portfolio of twenty to thirty stocks would average out to near zero. This is the mathematical content of the principle of diversification. A practical implication follows from the magnitude of \(R^2 \approx 0.34\): only about one-third of NVDA’s daily volatility comes from broad market moves, and the remaining two-thirds reflects firm-specific news flow. That residual two-thirds is exactly the variation that a stock-picker is paid to forecast — the space in which active equity research and alternative-data signals attempt to add value.
Inference for Regression Coefficients
Inference on Regression Coefficients
Background: Fisher, Neyman-Pearson, and the p-Value Wars
The significance test is one of the most misunderstood tools in science. It has two origins, often conflated. Ronald A. Fisher (1890–1962) introduced the \(p\)-value as a continuous measure of evidence: a small \(p\) suggests the data are incompatible with the null hypothesis, and the researcher should update their beliefs accordingly. Fisher never advocated for a hard 0.05 threshold — he saw it as one useful benchmark among many.
Jerzy Neyman and Egon Pearson (1933) proposed a different framework: binary decision-making with explicit control of Type I error (false positive rate \(\alpha\)) and Type II error (false negative rate \(\beta\)). Their approach requires specifying \(H_0\) and \(H_a\) before seeing the data and then making a binary reject/fail-to-reject decision. This is the framework behind “significance at the 5% level.”
Modern practice unhelpfully blends both frameworks. Researchers compute Fisher’s \(p\)-value but make Neyman-Pearson binary decisions — a hybrid neither inventor endorsed.
The replication crisis in social science (2010s) exposed the dangers of misusing \(p\)-values. Studies with \(p < 0.05\) failed to replicate at rates exceeding 50% in psychology and 30% in economics. The causes were multiple: \(p\)-hacking (testing many hypotheses and reporting only significant ones), underpowered studies (small \(n\)), and confusion about what \(p\) actually means.
In finance, the replication crisis is real too. Harvey, Liu and Zhu (2016, Review of Financial Studies) surveyed 316 claimed “factors” in the equity premium literature and concluded that the \(t\)-statistic threshold for declaring a new factor significant should be raised from 2.0 to at least 3.0, given the multiple testing problem. The lesson for CAPM: a \(p\)-value is evidence, not proof.
What a \(p\)-value is NOT: It is NOT the probability that the null hypothesis is true. It is NOT the probability that your result occurred by chance. It IS the probability of observing data this extreme (or more extreme) if \(H_0\) were true. These distinctions are not pedantic — confusing them leads to systematic over-confidence in regression results.
LINE Assumptions for Inference
For the \(t\)-tests and confidence intervals reported by statsmodels to be valid, the regression must satisfy four assumptions, traditionally summarised by the acronym LINE. Linearity (L) requires that the conditional mean of \(Y\) given \(X\) truly lies on a straight line, \(E[Y \mid X] = \alpha + \beta X\). Independence (I) requires that the error terms \(\epsilon_i\) across observations be uncorrelated with one another. Normality (N) requires that each \(\epsilon_i\) be drawn from a normal distribution with mean zero and some constant variance. Equal variance, or homoscedasticity (E), requires that the variance of \(\epsilon_i\) does not depend on \(x_i\). When all four assumptions hold, the estimators \(\hat{\alpha}\) and \(\hat{\beta}\) follow \(t\)-distributions, reported \(p\)-values can be trusted at face value, and confidence intervals have their nominal coverage probability. When any assumption is violated, the point estimates may still be informative, but the \(p\)-values and confidence intervals may be misleading.
Sampling Distributions of \(\hat{\alpha}\) and \(\hat{\beta}\)
The OLS estimates are themselves random variables: a different sample from the same population would yield a different pair \((\hat{\alpha}, \hat{\beta})\). To quantify how variable these estimates are, define the residual variance \(s^2 = SSE / (n - 2)\) and the sum of squared deviations of the predictor \(S_{xx} = \sum_i (x_i - \bar{x})^2\). The standard errors of the slope and intercept estimators are then given by
\[ SE(\hat{\beta}) = \frac{s}{\sqrt{S_{xx}}} \qquad SE(\hat{\alpha}) = s\,\sqrt{\dfrac{1}{n} + \dfrac{\bar{x}^2}{S_{xx}}}. \]
The standard error of \(\hat{\beta}\) shrinks when the predictor has a wide spread (large \(S_{xx}\)), when the residuals are tight (small \(s\)), and when the sample is large. A wider design — more variation in \(x\) — gives the estimator more leverage and produces a sharper estimate of the slope.
Why Do Standard Errors Matter?
Coefficients describe the relationship; standard errors describe the trust that can be placed in that description. Every \(t\)-statistic in the regression output is the ratio of a coefficient to its standard error, \(t_{\hat{\alpha}} = \hat{\alpha} / SE(\hat{\alpha})\) and \(t_{\hat{\beta}} = \hat{\beta} / SE(\hat{\beta})\), and every confidence interval is the point estimate plus or minus a \(t\) critical value times the standard error, \(\hat{\beta} \pm t_{0.025} \cdot SE(\hat{\beta})\). Standard errors therefore drive both the binary significance test and the continuous interval estimate. A coefficient without its standard error is half a story: the point estimate gives a centre, but the standard error gives the margin of error that surrounds it, and without that margin the analyst cannot tell whether a sizeable apparent effect is genuine or a sampling artefact.
model.summary() Output
The statsmodels summary table reports the following row for each coefficient:
| coef | std err | t | P>|t| | [0.025, 0.975] | |
|---|---|---|---|---|---|
| const (\(\alpha\)) | 0.0036 | 0.001 | 2.706 | 0.007 | 0.001, 0.006 |
| Mkt_excess (\(\beta\)) | 2.3320 | 0.166 | 14.038 | 0.000 | 2.005, 2.659 |
Each row implicitly conducts a two-tailed hypothesis test of the form \(H_0: \theta = 0\) versus \(H_a: \theta \ne 0\), where \(\theta\) stands for the coefficient on that row. The t column reports the test statistic; the P>|t| column reports the corresponding two-tailed \(p\)-value. The bracketed pair at the right is the lower and upper end of the ninety-five per cent confidence interval. Reading across the row, the analyst can simultaneously assess whether the coefficient is significantly different from zero (small \(p\)-value), how precisely it is estimated (narrow confidence interval), and what plausible range of values is consistent with the data.
\(t\)-Tests for Slope and Intercept
Under the LINE assumptions, each standardised estimate follows a \(t\)-distribution with \(n - 2\) degrees of freedom. The slope test takes the form
\[ t_{\hat{\beta}} = \frac{\hat{\beta}}{SE(\hat{\beta})} \;\sim\; t_{n-2} \qquad H_0: \beta = 0 \]
and a \(p\)-value below 0.05 leads the analyst to reject the null and declare \(x\) a significant predictor of \(y\). The intercept test is structurally identical,
\[ t_{\hat{\alpha}} = \frac{\hat{\alpha}}{SE(\hat{\alpha})} \;\sim\; t_{n-2} \qquad H_0: \alpha = 0, \]
and is read the same way. In the CAPM, these two tests answer two very different business questions. The slope test asks whether the market explains any portion of NVDA’s return at all, and a non-significant slope would mean that NVDA’s exposure to broad market moves is statistically indistinguishable from zero. The intercept test asks whether NVDA earns a free lunch above and beyond what its market exposure justifies — whether, in the language of finance, it generates a statistically detectable Jensen’s alpha.
What Is a \(p\)-Value? (Two-Tailed)
A two-tailed \(p\)-value is the probability of observing a test statistic at least as extreme as the one actually observed, assuming the null hypothesis is true. It is, in other words, a measure of how surprising the data would be under \(H_0\). The conventional decision rule rejects \(H_0\) when \(p < 0.05\) and fails to reject otherwise, although this threshold is a convention rather than a deep statistical truth. As a concrete illustration, a \(p\)-value of 0.003 for the estimated slope \(\hat{\beta}\) implies that, were the true \(\beta\) equal to zero, only three samples in a thousand would produce an estimate this large or larger in absolute value — sufficient evidence to conclude that the predictor genuinely matters. A common misreading is to interpret the \(p\)-value as the probability that \(H_0\) is true; it is not. The \(p\)-value conditions on \(H_0\) being true and reports the tail probability of the data, not the posterior probability of the hypothesis.
One-Tailed \(p\)-Value
In many applied settings the analyst wants to test a directional hypothesis rather than the symmetric two-sided alternative. A right-tailed test takes the form \(H_0: \beta \le 0\) against \(H_a: \beta > 0\), asking whether \(\beta\) is positive. When the observed \(t\)-statistic is positive and consistent with the alternative, the one-tailed \(p\)-value is exactly half the two-tailed \(p\)-value; when the observed \(t\)-statistic has the wrong sign relative to the alternative, the one-tailed \(p\)-value is \(1 - p_{\text{two}}/2\). A left-tailed test, \(H_0: \beta \ge 0\) versus \(H_a: \beta < 0\), follows the same logic with the roles of positive and negative reversed. The practical rule is that statsmodels always reports the two-tailed \(p\)-value, and the analyst should divide by two only when the sign of the \(t\)-statistic actually points in the direction of the alternative hypothesis.
Testing \(\alpha > 0\) and \(\beta > 1\) in CAPM
Two natural directional questions arise in the CAPM context. The first asks whether NVDA earns a positive Jensen’s alpha — that is, whether \(H_0: \alpha \le 0\) can be rejected in favour of \(H_a: \alpha > 0\). Because the estimated alpha is positive, the one-tailed \(p\)-value is exactly half the two-tailed value reported by statsmodels, and a value below 0.05 is sufficient to declare the alpha statistically positive. The first code block below carries out that calculation directly from the existing fitted model.
The second question is structurally different. To ask whether NVDA is more aggressive than the market — whether \(H_0: \beta \le 1\) can be rejected against \(H_a: \beta > 1\) — the default \(t\)-statistic in statsmodels will not suffice, because that statistic tests the null \(\beta = 0\). The analyst must recompute the test statistic relative to the new null, \(t = (\hat{\beta} - 1) / SE(\hat{\beta})\). The second code block performs this shift and compares the resulting \(t\) to a one-tailed critical value.
The general principle is that the default statsmodels test is always against \(\beta = 0\), the question of whether the predictor matters at all. To test against any threshold other than zero, the analyst recentres the test by subtracting the threshold from the estimated coefficient before dividing by the standard error.
Critical Value Approach
The \(p\)-value approach has an equivalent dual called the critical value approach. Rather than computing the tail probability of the observed test statistic and comparing it to 0.05, the analyst computes the critical value \(t_{\text{crit}}\) from the relevant \(t\)-distribution and rejects the null when the absolute value of the observed \(t\) exceeds it. For a two-tailed test of \(H_0: \beta = 0\) at the five per cent significance level, the rejection rule is \(|t| > t_{0.025, n-2}\). For a one-tailed right test of \(H_0: \beta \le 0\), the rule is \(t > t_{0.05, n-2}\). The two approaches always reach the same conclusion because they are algebraically dual. For large samples, the critical values converge to the familiar standard-normal benchmarks of 1.96 for a two-tailed test and 1.645 for a one-tailed test, which is why these numbers appear so often in applied statistics.
Confidence Intervals for \(\alpha\) and \(\beta\)
A ninety-five per cent confidence interval is constructed so that, in repeated sampling from the same population, ninety-five per cent of such intervals would contain the true parameter value. The formula combines the point estimate, a \(t\) critical value, and the standard error of the estimator.
\[ \text{CI for } \alpha:\quad \hat{\alpha} \;\pm\; t_{0.025,\,n-2} \cdot SE(\hat{\alpha}) \]
\[ \text{CI for } \beta:\quad \hat{\beta} \;\pm\; t_{0.025,\,n-2} \cdot SE(\hat{\beta}) \]
The reported intervals capture the precision of each estimate. The ninety-five per cent confidence interval for \(\beta\), approximately \([2.02, 2.42]\), says that NVDA’s market sensitivity is plausibly anywhere between 2.02 and 2.42. The corresponding interval for \(\alpha\), approximately \([0.0024, 0.0063]\), does not contain zero, confirming that the intercept is significantly different from zero at the five per cent level.
The interval for \(\beta\) admits a precise interpretation. If the same estimation were repeated on one hundred independent four-hundred-day samples drawn from the same process, approximately ninety-five of the resulting intervals would contain the true \(\beta\). The particular interval at hand does not contain 1.0, which provides statistical confirmation that NVDA is more aggressive than the market: the null \(\beta = 1\) can be rejected at the five per cent level. Translating to capital exposure, the width of the interval — roughly 0.40 — implies that on a ten per cent market decline NVDA’s expected loss could plausibly range from about minus 20.2 to minus 24.2 per cent. That estimation uncertainty is what options traders price into any NVDA derivative position, and it is why options market makers track standard errors and not merely point estimates. The interval for \(\alpha\) excluding zero is similarly noteworthy: in this simulated dataset, NVDA has a statistically significant positive Jensen’s alpha — by construction, since the simulation injected one. In real markets such findings are rare and transient, because any genuine free lunch attracts arbitrage capital that quickly drives the alpha back toward zero — the empirical content of the efficient market hypothesis.
A subtle but important misreading must be flagged. A student seeing a vanishingly small \(p(\alpha)\) might be tempted to conclude that there is, say, a 1-in-1000 chance NVDA’s true alpha is zero. This reading is wrong. The \(p\)-value assumes \(H_0\) is true and computes the probability of observing data this extreme; it does not provide a posterior probability over hypotheses, which would require a Bayesian framework. The correct interpretation is that if the true alpha were zero, only a tiny fraction of samples would produce an estimated alpha this large or larger in magnitude.
Reading the Statsmodels CI Output
The model.summary() call returns a single table containing every quantity discussed so far:
| coef | std err | t | P>|t| | [0.025 | 0.975] | |
|---|---|---|---|---|---|---|
| const (\(\alpha\)) | 0.0043 | 0.001 | 4.32 | 0.000 | 0.0024 | 0.0063 |
| Mkt_excess (\(\beta\)) | 2.221 | 0.103 | 21.65 | 0.000 | 2.020 | 2.422 |
Each column has a fixed meaning. The coef column reports the point estimate \(\hat{\alpha}\) or \(\hat{\beta}\). The std err column reports the standard error of that estimate. The t column is the ratio of the coefficient to its standard error, equivalent to the test statistic for the null of zero. The P>|t| column is the corresponding two-sided \(p\)-value. The bracketed range at the right reports the lower and upper endpoints of the ninety-five per cent confidence interval. There is a logical equivalence between the \(p\)-value and the confidence interval at the same significance level: the interval contains zero if and only if the \(p\)-value exceeds 0.05. The two columns carry identical information, but reading both is a useful habit because the interval communicates the magnitude of uncertainty in addition to its mere presence.
Exercise: Inference on CAPM
Using the CAPM model on NVDA data:
- What is the 95% CI for \(\beta\) (market sensitivity)?
- Can you reject \(H_0: \beta = 1\) at 5% significance? Hint: \(t = (\hat{\beta} - 1) / SE(\hat{\beta})\). Compare to \(t_{0.025, n-2} \approx 1.96\).
- What does the CI for \(\alpha\) tell you about NVDA’s risk-adjusted performance?
- Re-run for TSLA. Compare \(\beta_{\text{TSLA}}\) vs \(\beta_{\text{NVDA}}\) — which is more aggressive?
Confidence vs. Prediction Intervals and Train/Test
Confidence and Prediction Intervals
Background: Uncertainty About Means vs. Uncertainty About Individuals
The distinction between a confidence interval for a mean and a prediction interval for an individual observation is one of the most important — and most frequently confused — concepts in applied statistics. It maps directly onto a fundamental business distinction: are you making a portfolio decision (about the average across many instances) or a single-bet decision (about one specific case)?
A mutual fund manager asking “what is NVDA’s average return on strong market days?” needs a CI for \(\mu_Y\). As the fund holds NVDA for thousands of trading days, the law of large numbers causes the realized average to converge toward the true mean — the individual noise cancels out. The CI for \(\mu_Y\) narrows toward zero as the sample size grows: in the limit, the manager learns the true conditional mean exactly.
A risk manager asking “what is NVDA’s worst likely loss tomorrow?” needs a prediction interval. Tomorrow is a single observation. No amount of historical data eliminates the irreducible randomness of \(\epsilon_{t+1}\). The PI never collapses — it is bounded below by \(\pm z_{\alpha/2} \cdot \sigma\), the individual observation standard deviation, no matter how large \(n\) grows.
This asymmetry matters enormously for financial risk management. Value-at-Risk (VaR), the standard risk measure at banks, is essentially a quantile of the prediction interval — it estimates the loss that will not be exceeded with 95% (or 99%) probability on a single future day. Using the CI for \(\mu_Y\) instead of the PI would grotesquely understate the risk, producing intervals ten times too narrow.
The same distinction appears in operations and marketing. A retailer predicting demand for a product category across hundreds of stores should use CI for \(\mu\). A retailer predicting demand at a single new store — used to decide its inventory position — should use the PI.
Healthcare analytics provides a clear example. A hospital administrator estimating the average length of stay for knee replacement patients (to plan staffing levels) should use the CI for \(\mu_Y\). A surgeon telling this specific patient how long they will likely stay uses the PI. The PI is always wider — because a single patient has their own idiosyncratic biology, not just the average.
Two Questions, Two Intervals
Given a new predictor value \(x^*\) — for instance, a market return of plus one per cent — the analyst can pose two structurally different questions, each answered by a different interval. The first is a question about a population mean: across all days on which the market returns \(x^*\), what is the average NVDA return? The quantity of interest is the conditional mean \(\mu_Y = \alpha + \beta x^*\), and the confidence interval for \(\mu_Y\) captures the uncertainty in estimating that mean. This interval is appropriate whenever the decision in question concerns a population average — a long-run benchmark, the expected value of many similar bets, or the policy-relevant effect of a treatment on a population. The second question concerns a single future realisation rather than an average. Given that the market returns \(x^*\) tomorrow specifically, what will NVDA return on that one day? The quantity of interest is the random variable \(Y = \mu_Y + \epsilon\), which embeds both the uncertainty in the mean and the irreducible idiosyncratic noise. The prediction interval for \(Y\) is wider than the confidence interval for \(\mu_Y\) because it must accommodate the noise term, and it is the right tool whenever the decision concerns a specific future observation rather than a population summary.
The Formulas — One Extra Term
Both intervals are centred at the same point estimate, \(\hat{y}^* = \hat{\alpha} + \hat{\beta} x^*\), and both use the same critical value from the \(t\)-distribution. They differ in the term under the square root that constitutes the standard error. The confidence interval for the mean takes the form
\[ \hat{y}^* \pm t_{\alpha/2,\,n-2} \cdot s\sqrt{\dfrac{1}{n} + \dfrac{(x^*-\bar{x})^2}{S_{xx}}}, \]
while the prediction interval for a single new observation adds a unit term inside the square root:
\[ \hat{y}^* \pm t_{\alpha/2,\,n-2} \cdot s\sqrt{\color{red}{1} + \dfrac{1}{n} + \dfrac{(x^*-\bar{x})^2}{S_{xx}}}. \]
The extra “1” inside the radical accounts for the individual error term \(\epsilon\) of the new observation. Even if the conditional mean \(\mu_Y\) were known with perfect precision, a single observation would still vary around that mean with standard deviation \(\sigma\). The prediction interval is therefore always wider than the confidence interval, and the gap between them does not vanish even as the sample size grows large.
Why PI Is Always Wider
A graphical depiction makes the asymmetry vivid. The confidence band for \(\mu_Y\) hugs the fitted regression line tightly, narrowing further as additional data is added, because the only source of uncertainty it captures is the precision with which the mean function has been estimated. The prediction band is much wider, because in addition to that estimation uncertainty it must accommodate the irreducible scatter of individual observations around the line. As the sample size grows large, the confidence interval for \(\mu_Y\) collapses toward the regression line itself: in the limit, the conditional mean is known with arbitrary precision. The prediction interval never collapses to a line, because the individual noise term has its own variance \(\sigma^2\) that no amount of data can eliminate.
Python: get_prediction() for Both Intervals
The two columns returned by summary_frame answer two different questions. The mean_ci columns give the confidence interval for \(\mu_Y\), which on a plus one per cent market day might appear as approximately \([+1.7\%, +2.1\%]\) — a narrow band around NVDA’s expected average return on such days. The obs_ci columns give the prediction interval for a single new observation, which for the same scenario might appear as \([-1.1\%, +4.9\%]\) — roughly fifteen times wider, because individual returns remain noisy even when their conditional mean is estimated precisely.
The output thus contains two fundamentally different statements about the same forecast. The confidence interval for the mean implies that across all historical days when the market gained exactly one per cent, the average NVDA return falls in the narrow 0.4 percentage point band. That precision reflects the four hundred training observations that have collectively pinned down the conditional mean. The prediction interval for a single new observation acknowledges that tomorrow specifically, with its own news flow and idiosyncratic shocks, could land anywhere in a band roughly six percentage points wide. That width captures the residual standard deviation of approximately two per cent per day, a quantity that cannot be reduced by collecting more historical data. The dramatic asymmetry between the two intervals — roughly a factor of fifteen in width — illustrates why risk management and strategy evaluation, despite working with the same model, require entirely different interval types. A portfolio manager who attempted to size daily risk based on the confidence interval for the mean would understate exposure by an order of magnitude.
In financial reporting and business analytics, the confidence interval for \(\mu_Y\) is sometimes presented as if it were a forecast range for a single future observation. This conflation is a serious error that systematically understates uncertainty. The discipline is to ask, before constructing or interpreting any interval, whether the decision concerns the mean of many outcomes or a single specific future value.
Plot Both Bands Together
Business Interpretation
The choice between the two intervals follows the decision context. The confidence interval for \(\mu_Y\) is the right tool whenever the analyst is estimating an average effect — for instance, when benchmarking a stock’s typical return on days the market is up by a specified amount, or when evaluating the expected impact of a policy change across a population. The prediction interval is the right tool whenever the decision concerns a specific future observation: sizing the worst-case loss on a single trading day, planning inventory for one specific period, or quoting a forecast range for a single new customer. The two questions sound similar in casual conversation but produce intervals of dramatically different width, and using the wrong one is a common and costly mistake.
If the market drops 2% tomorrow, use get_prediction() to find: (a) the 95% CI for NVDA’s expected return, and (b) the 95% PI for NVDA’s actual return. Which interval would a risk manager care about?
Train/Test Split
Background: The Out-of-Sample Discipline
The practice of withholding data for out-of-sample evaluation has its roots in a 1974 paper by statistician Mervin Stone, who formalized cross-validation as a method for selecting statistical models. Stone’s insight was simple but profound: a model’s performance on the data used to fit it is a biased estimate of performance on new data. The bias is called overfitting, and it grows with model complexity.
The concept became the cornerstone of machine learning. Every production ML system at Google, Amazon, and every quantitative hedge fund withholds a held-out test set that the model never sees during training. The test performance is the only performance metric that matters for deployment. The training performance is essentially irrelevant beyond diagnosing underfitting.
In financial econometrics, out-of-sample testing has special importance. The look-ahead bias in finance — using future information to fit a model and then reporting in-sample performance as a “forecast” — is one of the most pervasive sources of spurious results in the academic factor investing literature. Papers reporting “strategies” that earn 20% annualized returns in-sample routinely fail to deliver out-of-sample once transaction costs and data snooping are accounted for.
Harvey, Liu and Zhu (2016) surveyed 316 factors reported in the literature and estimated that roughly half of them are false discoveries. The discipline of train/test split — running the exact same code on held-out data before claiming any predictive result — is the minimum bar for credibility in quantitative finance.
For time-series data like stock returns, there is an additional constraint: you must not shuffle the data. Unlike cross-sectional data, financial returns have time structure. Using 2024 data to train a model and evaluating it on 2020 data is not just poor practice — it is a direct form of look-ahead bias that renders the entire exercise meaningless. Always train on earlier dates and test on later dates.
Every systematic trading strategy at Renaissance Technologies, Two Sigma, and Citadel goes through rigorous out-of-sample backtesting before any capital is allocated. “In-sample fit” is not a metric — it is expected to be good by construction. Only out-of-sample Sharpe ratio, maximum drawdown, and hit rate determine whether a strategy moves to production.
Why Split the Data?
A model’s in-sample performance is, by construction, an optimistic estimate of how well it will do on new data. The estimation procedure has already used the in-sample observations to choose coefficient values that minimise the sum of squared residuals on precisely that sample. To form an honest assessment of how the model will perform on observations it has not seen, the analyst must withhold a portion of the data from training and use it only for evaluation. The standard convention is an eighty-twenty split: estimate \(\alpha\) and \(\beta\) on the first eighty per cent of the data, then use the remaining twenty per cent to compute out-of-sample \(R^2\) and root-mean-squared error.
For financial time series, the split must respect the temporal ordering: the training set consists of earlier dates and the test set consists of later dates. Shuffling the rows before splitting would leak future information into the training set — a textbook case of look-ahead bias — and would render any subsequent out-of-sample test meaningless.
R\(^2\) and RMSE
Two summary statistics dominate out-of-sample evaluation. The coefficient of determination,
\[ R^2 = 1 - \frac{\sum (y_i - \hat{y}_i)^2}{\sum (y_i - \bar{y})^2}, \]
measures the fraction of total variance that the model explains. An \(R^2\) of one corresponds to perfect prediction; an \(R^2\) of zero corresponds to a model that performs no better than always predicting the sample mean of \(y\); negative values arise out-of-sample when the model is so misspecified that it predicts worse than the in-sample mean. The root-mean-squared error,
\[ \text{RMSE} = \sqrt{\frac{1}{n}\sum_{i=1}^{n} (y_i - \hat{y}_i)^2}, \]
reports the typical magnitude of a prediction error in the original units of \(y\). The denominator is \(n\), not \(n-2\): out of sample we are averaging squared errors over the test points, not estimating a residual variance from a model that consumed two degrees of freedom in training. (The in-sample residual standard error reported by statsmodels uses \(\sqrt{\text{SSE}/(n-k-1)}\) — a different quantity, designed for inference rather than prediction.) For daily equity returns the RMSE is typically around half a per cent. The two metrics carry complementary information: \(R^2\) is a unit-free proportion that allows comparison across datasets, while RMSE is an interpretable magnitude that allows the analyst to translate prediction error into dollar terms or basis points. Best practice reports both.
Compute Train and Test Metrics
If train and test R\(^2\) are close, the model generalises well — the CAPM beta estimated on past data is stable.
The two pairs of numbers from the comparison code merit a careful reading. A train \(R^2\) of approximately 0.34 paired with a test \(R^2\) of approximately 0.32 is exactly what an analyst hopes to see: the model’s ability to explain variance is stable across the two time periods. For a CAPM with a single predictor and a parsimonious financial relationship, overfitting is not a serious concern. A train RMSE of approximately 0.020 paired with a test RMSE of approximately 0.021 confirms the same story in absolute units — the typical daily prediction error is around two per cent, only marginally higher on the test set than on the training set, as one would expect by construction. Translated to a $10{,}000 position, a two per cent daily RMSE corresponds to a typical prediction error of about $200 per day. A large train-test gap, by contrast, would signal overfitting: a model with too many predictors that has memorised idiosyncratic noise in the training data without learning generalisable patterns. With CAPM’s single predictor the model is too simple to overfit. The risk runs the other way: the model may be underfit, leaving systematic patterns unexplained, which is precisely the motivation for extending the regression to several predictors at once in the multiple-regression section that follows.
Time-series rule: For stock returns, always sort by date and train on earlier data. If you shuffle before splitting, you will leak future information into the training set (look-ahead bias), and your test performance will be artificially inflated.
Overfitting vs. Underfitting
Two opposing pathologies haunt every modelling exercise. Underfitting arises when the chosen model is too simple to capture the systematic structure in the data. The training \(R^2\) is low and the test \(R^2\) is similarly low, because there is no genuine pattern that the analyst has learned and could project onto new data. The fix for underfitting is to add predictors, polynomial terms, or interaction effects. Overfitting is the opposite pathology: the chosen model is so flexible that it has memorised the noise in the training data, achieving a high training \(R^2\) but a substantially lower test \(R^2\) because the patterns it has learned do not generalise. The fix for overfitting is to remove predictors, regularise the coefficients using shrinkage methods, or otherwise constrain the model’s effective complexity.
| Underfitting | Overfitting | |
|---|---|---|
| Cause | Model too simple | Model too complex |
| Train R\(^2\) | Low | High |
| Test R\(^2\) | Also low | Much lower than Train |
| Fix | Add predictors / nonlinear terms | Remove predictors / regularise |
The visual signature of each regime is intuitive. An underfit model corresponds to a near-flat line passing through a cloud of data that clearly has slope: the model misses the trend. A well-fit model traces a smooth function that captures the systematic variation without chasing individual points. An overfit model produces a wiggly curve that contorts itself to pass close to every training observation but extrapolates wildly outside the observed range.
Detecting Overfitting: Train vs. Test Gap
A well-known graphical signature characterises the overfitting problem. As model complexity grows, training error declines monotonically — every additional parameter gives the model one more degree of freedom with which to fit the noise as well as the signal. Test error, by contrast, follows a U-shape: it falls as long as additional parameters are capturing genuine structure, then rises as additional parameters begin to overfit the training noise. The minimum of the test-error curve identifies the right level of complexity for the problem at hand. Three warning signs reliably indicate that a fitted model has crossed into overfitting territory: a train \(R^2\) substantially higher than the test \(R^2\), an observation that adding more variables improves training fit but degrades test performance, and the presence of many insignificant predictors that the analyst is reluctant to remove. The goal in every applied setting is the sweet spot: a model complex enough to capture genuine patterns but simple enough to generalise. Adjusted \(R^2\), AIC, BIC, and cross-validation are the standard tools for navigating that trade-off.
Diagnostics and Assumptions
Assumptions and Residual Analysis
Background: The Gauss-Markov Theorem and What It Guarantees
The theoretical foundation of OLS lies in the Gauss-Markov theorem (1900, formalized by David Hilbert): under the four LINE assumptions, OLS produces the BLUE estimator — Best Linear Unbiased Estimator. “Best” means lowest variance among all linear unbiased estimators. This is not a trivial guarantee: it says you cannot do better than OLS with linear estimators if the assumptions hold.
Breaking down what each assumption guarantees when violated:
L — Linearity: If \(E[\epsilon | x] \neq 0\), then \(\hat{\beta}\) is biased — it systematically over- or underestimates the true slope. The Gauss-Markov theorem fails entirely. No amount of data fixes a misspecified model; you are estimating the wrong thing. This is the most serious violation. Remedy: add polynomial terms, interaction terms, or use a nonlinear model.
I — Independence: Correlated errors (serial correlation in time series) mean the effective sample size is smaller than \(n\). Standard errors computed assuming independence are too small — the model appears more precise than it actually is. \(p\)-values are falsely optimistic. In financial returns, ARCH/GARCH volatility clustering is a direct violation of independence. Remedy: use robust standard errors (Newey-West HAC).
N — Normality: Non-normal residuals affect the distribution of \(t\) and \(F\) statistics, not their means. For large \(n\), the Central Limit Theorem saves you: \(\hat{\beta}\) is asymptotically normal regardless of \(\epsilon\)’s distribution. The concern is mainly in small samples. Fat tails in stock returns (which are well-documented) mean that confidence intervals may have less than their nominal 95% coverage.
E — Equal variance (homoscedasticity): Heteroscedasticity — where \(\text{Var}(\epsilon_i)\) depends on \(x_i\) — leaves \(\hat{\beta}\) unbiased but makes standard errors wrong. OLS is no longer BLUE (it is inefficient; Weighted Least Squares would do better). More practically, \(t\)-tests and confidence intervals are unreliable. Remedy: White’s robust standard errors (cov_type='HC3' in statsmodels).
This diagnostic framework is not just academic formalism. Before presenting any regression result to a board, a CFO, or a regulator, a competent analyst always runs residual diagnostics and reports the results — explicitly noting any violations and the remedies applied.
Heteroskedasticity in financial data: Almost all financial return series exhibit heteroscedasticity. Volatility in 2008 was far higher than in 2006. OLS using raw returns as \(y\) will produce standard errors that are wrong. The standard practice is to use HAC-robust standard errors (model.fit(cov_type='HAC', cov_kwds={'maxlags': 5})), which are consistent in the presence of both heteroscedasticity and autocorrelation.
The LINE Assumptions
A regression analysis is trustworthy in proportion to the degree to which its residuals \(e_i = y_i - \hat{y}_i\) behave like draws from an idealised error process. The four assumptions of that process — linearity (the conditional mean of \(\epsilon\) given \(x\) is zero), independence (residuals across observations are uncorrelated), normality (each residual is drawn from a normal distribution with mean zero), and equal variance (the residual variance is constant across observations) — collectively constitute the LINE conditions that justify the standard \(t\)-tests, \(F\)-tests, and confidence intervals. Financial return data routinely violates two of these assumptions. The independence assumption fails because of volatility clustering: large absolute returns tend to be followed by large absolute returns, producing a serial dependence in residual magnitudes that GARCH models attempt to capture. The normality assumption fails because daily equity returns are famously leptokurtic, exhibiting fat tails far heavier than the normal distribution predicts. The diagnostic plots that follow are designed to detect each of these violations.
Assumption L: Linearity — Residuals vs. Fitted
The linearity assumption is checked by plotting residuals against fitted values. Under a correctly specified linear model the residuals should appear as a random cloud centred around zero, with no systematic pattern. A curved arrangement — a U-shape or an inverted U — indicates that the linear specification has missed a nonlinear feature of the data, and that the conditional mean of \(y\) given \(x\) is genuinely a curve rather than a straight line. The remedy is to add polynomial terms in \(x\), transform the dependent variable (commonly via a logarithm), or switch to a model class capable of capturing the missing curvature.
Assumption I: Independence — Residuals Over Time
The independence assumption is checked by plotting residuals against the time index. Under independence the plot should show no discernible pattern: residuals fluctuate randomly around zero with no clustering and no systematic runs of same-sign deviations. In financial return data, two violations appear routinely. The first is volatility clustering, in which periods of large absolute residuals alternate with periods of small absolute residuals; this is the empirical phenomenon that GARCH models are designed to capture. The second is autocorrelation, in which residuals exhibit serial dependence in their sign. Either violation invalidates the standard error formulas, and the remedy is to use heteroscedasticity-and-autocorrelation-consistent (HAC) standard errors.
Assumption N: Normality — Q-Q Plot & Histogram
The normality assumption is checked using a quantile-quantile plot, which arrays each ordered residual against the theoretical quantile of a standard normal distribution at the same rank. Under normality the points lie on a straight diagonal line. The most common deviation in financial data is an S-shape, with points falling below the diagonal at the lower end and above it at the upper end, indicating that both tails of the residual distribution are heavier than the normal. Fat tails imply that extreme returns occur more often than the normal model predicts, which matters most in small samples. For large \(n\), the Central Limit Theorem ensures that the sampling distribution of the regression coefficients remains approximately normal even when the residuals themselves are not.
Assumption E: Equal Variance — Scale-Location Plot
The equal-variance, or homoscedasticity, assumption is checked using a scale-location plot, in which the absolute (or squared) residuals are arrayed against the fitted values. Under homoscedasticity the spread of the residuals should be roughly constant across the range of fitted values, producing a horizontal band of approximately uniform width. A funnel or fan shape, where the spread widens or narrows systematically with the fitted value, indicates heteroscedasticity — a violation of the equal-variance condition that biases the OLS standard errors and renders \(p\)-values and confidence intervals unreliable. Two remedies are standard. The analyst can switch to White’s heteroscedasticity-consistent standard errors (specifying cov_type='HC3' in statsmodels), or transform the dependent variable to stabilise the variance.
Checking Assumptions with Python
LINE Diagnostics — CAPM Output
The four-panel diagnostic plot arrays the four assumption checks in a single figure: residuals versus fitted in the top-left, residuals over time in the top-right, the Q-Q plot in the bottom-left, and a scale-location view of variance in the bottom-right. Each panel is read in isolation but assessed jointly with the others.
The top-left panel addresses linearity. A random scatter around the horizontal red line at zero confirms that the linear specification is adequate. A U-shape or inverted-U-shape suggests that the relationship between NVDA and SPY contains a nonlinear component, perhaps requiring a squared term or a regime-switching specification. For simulated CAPM data drawn from a linear data-generating process, this panel should appear clean. The top-right panel addresses independence by plotting residuals against the observation index. Runs of same-sign residuals or visible clusters of large residuals followed by clusters of small residuals indicate that independence has failed; in real daily return data this GARCH-type clustering is almost always present. The bottom-left panel addresses normality through the Q-Q plot. Points should follow the diagonal line; deviations at the extremes — typically S-shaped, with the lower tail below the line and the upper tail above — indicate the fat-tailed behaviour that characterises daily equity returns. The bottom-right panel addresses equal variance. A horizontal band of constant width confirms homoscedasticity; a funnel shape, in which the spread of the residuals grows with the fitted value, indicates heteroscedasticity. In the CAPM context, heteroscedasticity often appears because days of large market moves are also days of higher NVDA volatility.
Outliers, leverage, and influence are different things: An outlier has a large residual (far from the fitted line vertically). A high-leverage point has an unusual \(x\) value (far from \(\bar{x}\)). An influential point has large Cook’s distance — removing it would substantially change \(\hat{\beta}\). A point can be an outlier without being influential (if it is near the middle of the \(x\) range), and it can be influential without being an outlier (if it is extreme in \(x\) and happens to lie exactly on the regression line). Check Cook’s distance with model.get_influence().cooks_d for any analysis where individual observations might dominate the fit.
What Happens When Assumptions Fail?
| Violated | Effect on Inference | Consequence |
|---|---|---|
| L Linearity | \(\hat{\beta}\) is biased; model misses pattern | Predictions systematically wrong; tests test the wrong model |
| I Independence | SEs are wrong (usually too small) | \(p\)-values too optimistic; CIs too narrow |
| N Normality | \(t\)/\(F\)-tests are approximate | Unreliable in small samples; OK for large \(n\) (CLT) |
| E Equal var. | OLS inefficient; SEs biased | Wrong CI width; \(p\)-values unreliable |
L and I are the most damaging: they make the estimates themselves wrong or the uncertainty estimates wrong. Always check these first.
N and E are less severe for large \(n\) (CLT helps). Always check L and I first — they are the hardest to fix.
Assumptions Hold vs. Fail: The Big Picture
| Assumptions Hold | Assumptions Fail |
|---|---|
| \(\hat{\beta}\) is unbiased | L fails \(\Rightarrow\) \(\hat{\beta}\) biased |
| SEs are correct | I or E fails \(\Rightarrow\) SEs wrong |
| \(t\)-tests & \(F\)-tests are exact | N fails \(\Rightarrow\) tests approximate |
| CIs have correct coverage | CIs may under/over-cover |
| Predictions are optimal | OLS no longer best estimator |
OLS always gives numbers — but those numbers are only trustworthy if the assumptions approximately hold. Diagnostic plots let you check before trusting the output.
A Worked Example: NBA Player Stats Predict Wins
Regression is everywhere in modern sports analytics. Daryl Morey’s Houston Rockets transformed the NBA in the 2010s by regressing wins on shot location and discovering that the expected points per attempt from beyond the three-point line dominated almost every mid-range shot. The result was the three-point revolution: by 2024 the league was taking more than three times the volume of three-point attempts it took in 2010. Under the hood, this strategic shift was driven by linear regressions run on play-by-play data. Below is a tiny worked example using twelve simulated NBA player-game lines.
Reading the output: the coefficients on points and assists are positive — more scoring and more playmaking raise the estimated win probability. The coefficient on turnovers is negative — every additional turnover lowers the chance of winning, which matches basketball intuition. rebounds enters with a small coefficient that is not statistically distinguishable from zero in this twelve-row sample. With \(n = 12\) and four predictors the standard errors are wide, so only the predictors with the largest effect sizes — typically points and turnovers — are likely to show \(p\)-values below the conventional 0.05 threshold.
One caveat before moving on. The outcome team_won is binary, taking only the values 0 and 1, and the model above is a linear probability model rather than a logistic regression. Fitted values can fall outside \([0, 1]\), and the residuals are mechanically heteroskedastic. We use it here because the goal is to keep the toolbox of this chapter — OLS, \(t\)-tests, and model.summary() — intact. Linear probability models remain a legitimate teaching tool and are still used in applied work for marginal-effect interpretation. Logistic regression is the right tool when the goal is calibrated probability prediction, and it is covered in a later course.
Multiple Linear Regression
Multiple Linear Regression
Background: From One Predictor to Many
The move from CAPM (one predictor) to multiple regression parallels a general pattern in applied research. A single-predictor model is the natural starting point — it is easy to fit, easy to plot, and easy to interpret — but it almost always omits forces that genuinely shape the response. CAPM was dominant through the 1970s, and then systematic anomalies began to accumulate: stocks with low market capitalisation consistently outperformed CAPM predictions, and stocks trading at low price-to-book ratios did the same. The market factor alone could not explain these patterns, and the field moved to multi-factor models that extended the regression framework to several predictors at once.
The same story repeats outside finance. A pharmacy chain that wants to predict per-branch profit cannot rely on local income alone: the age distribution of the catchment area, the local birth rate, and the prevalence of chronic conditions all matter, and the multiple regression delivers a partial effect for each. A university that wants to predict final-exam performance cannot rely on study hours alone: attendance, prior GPA, sleep, and competing time commitments all enter the calculation, and the multiple regression assigns each a partial effect that is interpretable holding the others constant. In every setting, the regression framework gives the analyst a quantitative answer to the question “which of these candidate predictors carries genuine independent information about the response, and which is duplicated by another variable already in the model?”
The shift from single-predictor to multi-predictor models is one of the most consequential moves in applied analytics. Coefficients on a single-predictor model attribute all of the variation in the response to that one variable, including variation that the omitted predictors would have explained. Adding the omitted predictors typically shrinks the original coefficient — sometimes dramatically, sometimes even flipping its sign — and the new partial effect is the one that matters for decisions. This is why product managers, marketing analysts, and equity researchers alike spend considerable effort enumerating candidate predictors before they trust any single coefficient.
Why Do We Need Multiple Regression?
A simple linear regression uses a single predictor to explain a response variable. In practice, however, most outcomes of business or social-science interest are shaped by several systematic forces simultaneously. A pharmacy branch’s annual profit depends on local income, the age distribution of the catchment area, and the prevalence of chronic conditions — not on income alone. A student’s final-exam score depends on study time, attendance, prior preparation, and a handful of secondary habits — not on study time alone. When the analyst fits a single-predictor regression to data that is governed by many predictors, the resulting coefficient of determination remains low and a meaningful share of the residual variance can be attributed to predictors that the model has omitted. Beyond the loss of explanatory power, omitting a relevant variable creates the more serious problem of omitted variable bias: if a missing variable correlates with both the included predictor and the dependent variable, the estimated coefficient on the included predictor is biased — it captures both the direct effect and an indirect transmission through the omitted variable. Multiple linear regression solves the bias problem by including all relevant predictors simultaneously, giving each coefficient the interpretation of a partial effect, the change in the response associated with a one-unit increase in that predictor holding all the others constant.
Multiple Linear Regression (MLR)
Simple regression uses a single predictor; multiple regression generalises the framework to several predictors entering the model simultaneously, written as
\[ y = \alpha + \beta_1 x_1 + \beta_2 x_2 + \cdots + \beta_k x_k + \epsilon. \]
The intercept \(\alpha\) is the predicted value of \(y\) when every predictor equals zero. Each slope coefficient \(\beta_j\) is a partial slope, representing the change in \(y\) associated with a one-unit increase in \(x_j\) while the other predictors are held constant — the ceteris paribus interpretation that makes multiple regression the workhorse of applied econometrics. The error term \(\epsilon\) satisfies the same LINE assumptions as in the simple case. Conceptually, each predictor contributes its own weighted impact \(\beta_j x_j\) to the prediction, and the fitted value \(\hat{y}\) aggregates these contributions along with the intercept.
Interpreting Coefficients: “Ceteris Paribus”
The defining distinction between multiple and simple regression is the interpretation of each slope coefficient as a partial effect. Consider a model of house prices, \(\hat{y} = 50 + 0.1 \times \text{sqft} + 15 \times \text{bedrooms}\), where price is measured in thousands of dollars. The coefficient on square footage, 0.1, says that each additional square foot raises the predicted price by one hundred dollars, holding the number of bedrooms constant. The coefficient on bedrooms, 15, says that each additional bedroom raises the predicted price by fifteen thousand dollars, holding square footage constant. Without the “holding constant” qualifier, the slope coefficient in a multiple regression is generally different from the corresponding slope in a simple regression that uses only one predictor. Adding correlated predictors to a model can change every coefficient — including their signs — which is not a flaw but the intended feature: the procedure is correctly partitioning the explanatory power among the included variables and controlling for confounding influences.
Fitting MLR in Python
# General MLR: identical syntax -- just pass multiple columns
# y = b0 + b1*x1 + b2*x2 + ... + bk*xk
X = sm.add_constant(df[["x1", "x2", "x3"]]) # k predictors
model = sm.OLS(y, X).fit()
print(model.summary())
# Key outputs:
# model.params -> coefficients (beta_0, ..., beta_k)
# model.pvalues -> p-value for each beta_j
# model.rsquared_adj -> Adjusted R-squared
# model.conf_int() -> 95% CIs for all betasThe code is identical to SLR — just pass a DataFrame with \(k\) columns instead of one. OLS handles everything.
Live Demo: Predicting Pharmacy Profit
The pharmacy profit example provides a concrete business setting in which multiple regression is essential. A pharmacy chain operates 111 branches across major US metropolitan areas, and the analytic question is which local demographic characteristics best predict per-branch profitability. Six candidate predictors are available for each branch’s catchment area: median household income, disposable income, the birth rate per thousand, the social-security recipient density per thousand, the cardiovascular death rate per hundred thousand, and the share of the population aged sixty-five or older. The target variable is annual profit at the branch level. With six predictors and 111 observations, the dataset is well within the sample-size range where multiple regression provides reliable inference, and the predictors span the demographic dimensions most likely to influence prescription volumes and over-the-counter sales.
Real business case, small enough to read by eye, large enough to need multiple regression. Profit depends on multiple local features — no single one tells the full story.
Fit the Pharmacy Multiple Regression
After running, the analyst examines several elements of the summary output in sequence. The R-squared statistic reports overall fit, while the adjusted R-squared penalises the raw R-squared for the number of predictors and is the appropriate metric for comparing models of differing dimensionality. The coef column gives the direction and magnitude of each estimated effect, and the P>|t| column flags which features remain statistically significant once the others are controlled for. The F-statistic at the bottom of the table reports the joint significance of all predictors taken together, testing the null hypothesis that none of the demographic variables explains profit.
Interpreting the Pharmacy Model
- Positive coef = feature increases profit, holding others constant
- Negative coef = feature decreases profit
- Insignificant (p > 0.05) = not enough evidence this feature matters once others are accounted for
The pharmacy regression output delivers a quantitative portrait of which demographic forces shape per-branch profitability, and the coefficients reward an economically literate reading. The income coefficient is typically positive and significant: each additional thousand dollars of median household income adds a predictable dollar amount to expected branch profit, holding birth rate, social-security density, and age distribution constant. Higher-income neighbourhoods can afford to fill more prescriptions and to spend more on over-the-counter products, and this effect is identified as income’s independent partial contribution rather than a confounded reflection of correlated demographics. The coefficient on the share of the population aged sixty-five or older is generally positive, reflecting the well-documented fact that older populations fill more prescriptions per capita; a one-percentage-point increase in the elderly share has a definable dollar impact on branch profit even after controlling for income and birth rate. The birth-rate coefficient is more ambiguous: areas with high birth rates often have young, generally healthy populations with fewer chronic medications, and the sign and significance of this coefficient effectively test whether a younger demographic mix reduces pharmacy profitability — a hypothesis with direct site-selection implications. At the level of the model as a whole, the F-statistic tests joint significance: a \(p\)-value below 0.05 means at least one demographic feature genuinely predicts profit, and for this dataset the F-test is typically highly significant. Comparing the adjusted \(R^2\) to the raw \(R^2\) is the standard parsimony check; a substantial gap indicates that some predictors are adding noise without explanatory power, and dropping them tends to improve the adjusted value.
Multicollinearity in demographic data: Income, disposable income, and social security density are likely highly correlated across metro areas. This can inflate standard errors and make individual coefficients unstable. Always run VIF diagnostics on demographic regression models. If VIF > 10 for Income and Disposable Income, consider using only one of them (or constructing a composite).
Drop the insignificant features and refit. Does Adjusted R² go up or down? What does that tell you about model parsimony?
Predict Profit for a New Branch
Suppose a new branch opens in a city with these demographics. What does the model predict?
CI for mean — “If we opened many branches with these demographics, average profit would be in this range.” Used for portfolio decisions.
PI for individual — “This specific branch’s profit will be in this range.” Wider — more cautious for single-branch decisions.
R\(^2\) vs. Adjusted R\(^2\)
A persistent technical hazard in applied work is the use of raw \(R^2\) to compare models with different numbers of predictors. Because the OLS objective minimises the sum of squared errors, adding any additional predictor — even one that is statistically useless — can only weakly decrease \(SSE\) and therefore can only weakly increase \(R^2 = 1 - SSE/SST\). Raw \(R^2\) rewards complexity mechanically and therefore cannot detect overfitting. The adjusted \(R^2\) corrects this defect by introducing an explicit penalty for the number of predictors,
\[ \bar{R}^2 = 1 - \frac{(1-R^2)(n-1)}{n-k-1}. \]
Adjusted \(R^2\) can move in either direction when a predictor is added: it rises if the new predictor improves fit by more than the penalty, and it falls if the new predictor adds noise. The practical rule is that any comparison across models with different numbers of predictors must use adjusted \(R^2\), AIC, BIC, or out-of-sample \(R^2\) — never raw \(R^2\) alone.
Multicollinearity
When two or more predictors are strongly correlated with one another, OLS still produces valid estimates and predictions but the individual coefficients become unstable in a specific sense. Standard errors balloon, confidence intervals widen, individual coefficients may flip sign or change in magnitude when a single related variable is added or removed, and the model can present the unsettling combination of a high overall \(R^2\) alongside many individually insignificant \(p\)-values. The diagnostic for multicollinearity is the variance inflation factor,
\[ \text{VIF}_j = \frac{1}{1 - R_j^2}, \]
where \(R_j^2\) is obtained by regressing predictor \(x_j\) on all the other predictors. The intuition is straightforward: if \(x_j\) can be largely predicted from the other variables, then there is little independent information in \(x_j\) for the regression to use, and the standard error on its coefficient is inflated accordingly. A common rule of thumb treats VIF values below 5 as unproblematic, values between 5 and 10 as cause for caution, and values above 10 as severe. Crucially, multicollinearity does not bias the predictions of the model — overall fit, \(R^2\), and forecasts are unaffected. The damage is confined to the individual coefficient estimates, whose elevated uncertainty makes them unreliable as descriptions of partial effects.
Detecting Multicollinearity in Python
from statsmodels.stats.outliers_influence import (
variance_inflation_factor)
# Compute VIF for each predictor
X_vif = sm.add_constant(df[["x1", "x2", "x3"]])
for i, col in enumerate(["x1", "x2", "x3"]):
vif = variance_inflation_factor(X_vif.values, i + 1)
print(f"{col}: VIF = {vif:.2f}")Rule of thumb: If any VIF \(> 10\), consider dropping or combining the offending variable. VIF \(< 5\) is generally safe.
Understanding \(R_j^2\) in VIF — A Simple Example
A small example clarifies the relationship between \(R_j^2\) and VIF. Consider a regression with three predictors — age (\(x_1\)), income (\(x_2\)), and credit score (\(x_3\)). To compute the VIF for income, the analyst regresses \(x_2\) on the other two predictors, obtains the resulting \(R_2^2\), and computes \(1/(1 - R_2^2)\).
| Predictor | \(R_j^2\) | VIF | Status |
|---|---|---|---|
| \(x_1\) (age) | 0.10 | 1.11 | OK |
| \(x_2\) (income) | 0.85 | 6.67 | Warn |
| \(x_3\) (credit) | 0.82 | 5.56 | Warn |
A reading of \(R_j^2 = 0.85\) means that eighty-five per cent of the variation in income is already explained by age and credit score together, so income carries only fifteen per cent unique information for the regression — making it nearly redundant, with a VIF of 6.67 that flags moderate multicollinearity. A reading of \(R_j^2 = 0.10\) means that age is largely independent of the other predictors and contributes its own unique information, yielding a VIF close to one. The general principle is that high \(R_j^2\) values indicate that predictor \(j\) is well predicted by the others, producing an inflated standard error on its coefficient. The standard remedies are to drop one of the correlated predictors, to combine them into a composite index, or to use ridge regression, which introduces a small shrinkage penalty that stabilises the coefficient estimates at the cost of a small bias.
A Student-Performance Multiple Regression
The pharmacy demonstration showed multiple regression at work on cross-sectional business data. The next worked example shifts to a different domain — predicting student final-exam scores — for two reasons. First, the data-generating process is one every reader has lived through, which makes the coefficients easy to interpret without specialised background. Second, the five candidate predictors include both clearly useful inputs (hours of study, prior GPA) and inputs whose value is contested (sleep, extracurriculars), so the regression has genuine work to do in separating signal from noise. The response variable is the score on a 100-point final exam, and the five predictors describe a student’s typical weekly habits during the term:
study_hours_per_week— average hours spent on coursework outside classattendance_pct— fraction of lectures attended, expressed in per centprior_GPA— cumulative grade point average on a 4.0 scale entering the termsleep_hours_per_night— average sleep per night during the termextracurricular_hours_per_week— average hours per week on clubs, sports, and student organisations
The multiple regression model takes the form
\[ \text{final\_exam\_score} = \alpha + \beta_1 \text{study\_hours} + \beta_2 \text{attendance} + \beta_3 \text{prior\_GPA} + \beta_4 \text{sleep} + \beta_5 \text{extracurricular} + \epsilon. \]
Each coefficient is a partial effect: the change in expected final-exam score associated with a one-unit increase in that predictor, holding the other four constant.
What Each Predictor Means
The five candidate predictors and their expected effects on final-exam performance can be summarised compactly:
| Predictor | Unit | Expected sign | Intuition |
|---|---|---|---|
study_hours_per_week |
hours / week | \(+\) | More practice raises mastery |
attendance_pct |
per cent | \(+\) | Lectures cover testable material |
prior_GPA |
0.0–4.0 scale | \(+\) | Strong general ability transfers |
sleep_hours_per_night |
hours / night | \(+\) | Rested students consolidate learning |
extracurricular_hours_per_week |
hours / week | \(-\) or \(0\) | Time crowded out from study |
The first three predictors should clearly raise scores — more practice, more contact with instructors, and stronger background all support exam performance. The sign on sleep is positive in principle but small enough that it may not survive once study hours and prior GPA are controlled for. Extracurriculars are the variable in genuine doubt: they may enrich a student’s life without harming grades, or they may compete directly for the same scarce hours that study otherwise consumes. The regression’s job is to deliver a quantitative verdict.
Build the Multiple Regression
The code is identical to simple regression — just pass a list of five predictor columns. statsmodels handles everything.
The multiple-regression output is read the same way as the simple-regression output, but with five slopes instead of one. Each coefficient is a partial effect — the change in expected final-exam score associated with a one-unit increase in that predictor, holding the other four predictors constant. In the simulated draw, the three habit variables built directly into the data-generating process come through with the right signs and large \(t\)-statistics: each extra hour of weekly study adds roughly 1.8 points to the predicted exam score; each additional percentage point of attendance adds about 0.2 points; and each unit of prior GPA — a substantial increment on the 4.0 scale — adds roughly 12 points. The sleep coefficient sits around plus one point per hour, in line with the DGP but estimated with wider standard error because the day-to-day variation in sleep is narrow. The extracurricular coefficient is negative, approximately \(-0.3\) points per weekly hour, consistent with a modest opportunity-cost story rather than any catastrophic effect. The intercept of roughly 35 is the model-implied score for a hypothetical student who never studies, never attends class, enters with GPA zero, and never sleeps — an out-of-support combination, so the intercept matters only as a fitting constant, not as a literal forecast.
Interpreting the Coefficients
A representative draw of the simulated coefficients and \(p\)-values looks like the following. The point estimates fluctuate from run to run by sampling noise; the qualitative picture — three habit variables strongly significant with the expected positive sign, a smaller but identifiable sleep effect, and a negative extracurricular coefficient — is stable.
| Predictor | \(\hat{\beta}\) | p-value | Meaning |
|---|---|---|---|
| const (\(\alpha\)) | 35 | \(<0.001\) | Fitting constant — not literally interpretable |
study_hours_per_week |
\(+1.8\) | \(<0.001\) | Each extra hour adds 1.8 points |
attendance_pct |
\(+0.20\) | \(<0.001\) | Each extra % attended adds 0.2 points |
prior_GPA |
\(+12\) | \(<0.001\) | Strong background transfers to the exam |
sleep_hours_per_night |
\(+1.2\) | \(\sim 0.01\) | Modest but identifiable rest effect |
extracurricular_hours_per_week |
\(-0.3\) | \(\sim 0.03\) | Mild opportunity cost on study time |
Each \(\beta_j\): “holding the other predictors constant, a one-unit increase in this predictor changes the final-exam score by \(\beta_j\) points.”
The three habit variables that the DGP loads on most heavily — study hours, attendance, and prior GPA — clearly help exam performance. The sleep coefficient is positive and consistent with prior research on cognitive consolidation, but its magnitude is small enough that it could easily fail to clear the conventional 5% threshold in any particular sample. The extracurricular coefficient is the most policy-relevant: it is small, negative, and marginally significant. A student trading one hour of clubs for one hour of study would gain roughly \(1.8 - (-0.3) = 2.1\) exam points on average, a non-trivial effect but far from the dominant driver. The regression therefore supports nuanced advice: do not eliminate extracurriculars on the basis of this coefficient, but recognise that the time they consume is the single variable a student can most easily reallocate.
Testing Coefficient Significance
Individual \(t\)-test (same as SLR):
\[ H_0: \beta_j = 0 \quad\text{vs}\quad H_a: \beta_j \ne 0 \qquad t_j = \frac{\hat{\beta}_j}{SE(\hat{\beta}_j)} \;\sim\; t_{n-k-1} \]
Overall \(F\)-test (do any of the predictors carry information?):
\[ H_0: \beta_1 = \beta_2 = \cdots = \beta_k = 0 \qquad F = \frac{SSR/k}{SSE/(n-k-1)} \;\sim\; F_{k,\,n-k-1} \]
\(F\)-test significant \(\Rightarrow\) at least one predictor matters. Individual \(t\)-tests tell you which ones. Predictors with large \(p\)-values are candidates for dropping.
The F-statistic and the individual \(t\)-tests provide complementary information about model significance. The F-test addresses the joint hypothesis that every slope coefficient is zero. A large observed F with a tiny \(p\)-value rules out the possibility that none of the five habits matter, confirming that the model as a whole carries genuine explanatory power. The individual \(t\)-tests then proceed predictor by predictor. In the simulated draw shown above, study hours, attendance, and prior GPA are clearly significant by an overwhelming margin; sleep and extracurriculars sit close to the conventional 5% threshold and could easily land on either side of it from sample to sample. The economic interpretation of a borderline sleep coefficient is straightforward: the underlying true effect is modest, the within-sample variation in sleep is narrow, and the standard error inflates accordingly. A data-driven recommendation in a real research workflow would be to drop the weakly identified predictors and re-estimate the reduced model, then confirm using BIC or adjusted \(R^2\) that the smaller specification outperforms the full one — and to check that the conclusion is robust across multiple terms and student cohorts, not just one 300-student draw.
Multicollinearity Check
Two of the five predictors describe related student behaviours. Students who attend more classes also tend to be the same students whose prior GPA is high, so attendance_pct and prior_GPA may show non-trivial correlation. If their joint VIF exceeds 5, the model is asking each variable to do work that the other could partly do as well.
If VIF \(> 10\): drop or combine. Here all VIFs are low \(\Rightarrow\) multicollinearity is not a problem for this student model.
Interpretation. The VIF output for the five student-life predictors typically shows values close to 1.0, with attendance_pct and prior_GPA running a little higher than the rest because conscientious students tend to attend more and enter the term with a higher GPA. In the simulated DGP the predictors are independent by construction, so their VIFs hover near 1.0 by design; in real survey data, the correlation between attendance and prior GPA would push their VIFs into the 1.5–2.5 range, still well below the warning threshold of 5. A VIF that crossed 5 would not invalidate the predictions of the regression — multicollinearity inflates standard errors on individual coefficients without disturbing the model’s overall fit — but it would mean that the partition of credit between attendance and prior GPA becomes statistically unstable, and that small changes in the sample could swing those two coefficients in opposite directions.
VIF > 10 does not mean your predictions are wrong. Multicollinearity inflates standard errors on individual coefficients, making their signs unreliable and \(p\)-values misleading. But the model’s overall predictive accuracy (\(R^2\), RMSE, out-of-sample performance) is unaffected by multicollinearity. The pitfall is using a multicollinear model to interpret which predictor “matters most” — the answer changes dramatically when you drop one correlated variable.
Simple Regression vs. Multiple Regression: Does MLR Help?
Compare using Adjusted R\(^2\) (not R\(^2\), which always increases). If the 5-feature \(\bar{R}^2\) exceeds the 1-feature \(\bar{R}^2\), the extra predictors genuinely add explanatory power beyond study hours alone.
Interpretation. A representative summary of the two specifications looks like the following:
| Model | \(k\) | \(R^2\) | Adj \(R^2\) | What it means |
|---|---|---|---|---|
| study hours | 1 | 0.40 | 0.40 | Study hours alone explain 40 per cent of the variance |
| 5 features | 5 | 0.85 | 0.85 | Five habit predictors explain 85 per cent |
The 45-percentage-point improvement in adjusted \(R^2\) means that knowing a student’s attendance, prior GPA, sleep, and extracurricular load adds genuine predictive information beyond study hours alone. The improvement is not artifactual: adjusted \(R^2\) already penalises the four extra parameters. This is the quantitative justification for why a course-performance forecast that uses all available behavioural signals beats one based on study time alone.
The five-feature model also produces smaller residuals than the single-feature model — the idiosyncratic variation in exam scores after controlling for the five habits. A lower \(\hat{\sigma}\) means tighter prediction intervals and more precise statements about whether a particular student is over- or under-performing relative to expectations.
\(R^2\) always increases when you add variables, so never compare models on raw \(R^2\) alone. Use adjusted \(R^2\), AIC, or BIC. The table above shows that raw \(R^2\) goes from 0.40 to 0.85 (adding four predictors always improves raw \(R^2\)), and adjusted \(R^2\) confirms the improvement is genuine after penalising for the extra parameters.
CI for \(\mu_Y\) and Prediction Interval in MLR
The simple-regression formulas for the confidence and prediction intervals extend naturally to the multivariate case. For a new student described by the vector of predictor values \(\mathbf{x}^* = (x_1^*, \ldots, x_k^*)\), the confidence interval for the conditional mean \(\mu_Y \mid \mathbf{x}^*\) captures the average final-exam score across all students who share those exact characteristics; its width depends only on the standard error of the estimated conditional mean and shrinks toward zero as the sample size grows. The prediction interval for an individual future observation \(Y\) captures one particular student’s exam score given those same characteristics; it incorporates the additional individual variance \(\sigma^2\) and is therefore always wider than the confidence interval, with the gap between them remaining bounded below by the residual standard deviation regardless of how much historical data is available.
Variable Selection and Business Applications
Variable Selection
Background: The Information-Theoretic and Bayesian Foundations
Model selection — choosing which predictors to include — is one of the deepest problems in statistics. Two criteria dominate applied practice, each with a distinct theoretical foundation rooted in very different ideas.
AIC (Akaike Information Criterion) was introduced by Hirotugu Akaike in 1973, building on ideas from information theory and Kullback-Leibler divergence. Akaike’s insight was to frame model selection as an estimation problem: which model minimizes the expected information loss when approximating the true data-generating process? The resulting criterion penalizes the log-likelihood by \(2k\) (twice the number of parameters), trading off fit against parsimony. AIC is efficient: in large samples, the model minimizing AIC produces the lowest out-of-sample prediction error. It is the right criterion when your goal is forecasting.
BIC (Bayesian Information Criterion) was introduced by Gideon Schwarz in 1978 using a Bayesian argument. Schwarz asked: which model maximizes the marginal likelihood (the probability of the data under the model, averaged over the prior)? For a flat prior over models, Laplace approximation yields a penalty of \(k \ln n\) — growing with both the number of parameters and the sample size. BIC is consistent: as \(n \to \infty\), BIC selects the true model with probability approaching 1, provided the true model is in the candidate set. It is the right criterion when your goal is identifying the true model structure.
No criterion is simultaneously efficient and consistent — this is a fundamental statistical impossibility (the Yang 2005 impossibility theorem). The practical choice: when the goal is forecasting an outcome — a student’s exam score, a branch’s annual profit — minimize AIC. When the goal is to identify which predictors genuinely belong in the data-generating process, use BIC (which will typically favor a sparser model, penalizing the extra parameters more heavily).
The history of variable selection algorithms: Best subset selection examines all \(2^p\) subsets — feasible for \(p \le 30\) but computationally prohibitive for large \(p\). Forward stepwise selection was introduced as a greedy alternative in the 1960s; backward elimination followed. These greedy algorithms do not guarantee finding the globally optimal model, but they are fast and usually perform well in practice. The 2000s brought LASSO (Tibshirani 1996), which combines continuous variable selection with coefficient shrinkage via an L1 penalty — effectively solving the best subset problem as a convex optimization. The 2010s brought elastic net, group LASSO, and many ML extensions. The examples in this chapter implement the classical best-subset approach in plain Python with statsmodels; for large-\(p\) settings, LASSO (sklearn.linear_model.Lasso) is the modern benchmark.
At AQR and BlackRock’s factor research teams, model selection is never done purely by AIC/BIC on a single dataset. Economic intuition (is there a risk-based or behavioral story for this factor?), replication across different markets and time periods, and out-of-sample backtesting all inform which factors make it into a production model. Pure statistical selection from historical data without economic discipline is a recipe for data mining and false discoveries.
The Overall F-Test
The overall F-test addresses the question of whether the multiple-regression model has any explanatory power at all. The null hypothesis asserts that every slope coefficient equals zero — that none of the predictors carries a genuine relationship to the response — and the alternative asserts that at least one slope is non-zero. Formally,
\[ H_0: \beta_1 = \beta_2 = \cdots = \beta_k = 0 \quad\text{vs}\quad H_a: \text{at least one } \beta_j \neq 0, \]
and the test statistic compares the explained variance per parameter against the residual variance per remaining degree of freedom:
\[ F = \frac{SSR/k}{SSE/(n-k-1)} = \frac{R^2/k}{(1-R^2)/(n-k-1)} \;\sim\; F_{k,\,n-k-1}. \]
The F-test is in every model.summary(). An insignificant F (\(p > 0.05\)) means you cannot reject the joint null that all slopes are zero — i.e. the model as a whole has not demonstrated explanatory power in this sample. That is not the same as proving “no variable matters”: a single significant predictor may still exist but be masked by noise, by multicollinearity, or by an underpowered sample. Read an insignificant F as “the evidence here is weak,” not as “the variables are useless.”
Best Subset Selection
The goal of best subset selection is to identify, for each possible subset size \(k\) running from one to \(p\), the single combination of predictors that minimises the sum of squared errors. The algorithm proceeds in three steps. First, for each subset size \(k\), every one of the \(\binom{p}{k}\) possible combinations of predictors is enumerated, fit by OLS, and recorded. Second, for each \(k\), the model with the lowest SSE is retained, producing a list of \(p\) candidate models — the best model of each size. Third, a complexity-aware criterion such as AIC, BIC, adjusted \(R^2\), or Mallow’s \(C_p\) is used to choose among these candidates. A useful visual aid plots adjusted \(R^2\) for every candidate model against the model size \(k\) on the horizontal axis; the upper envelope of this scatter traces the frontier of best-of-size models, and the peak of the frontier identifies the optimal complexity for the dataset.
Best Subset Selection in Plain Python
The algorithm is short enough to write from scratch in 25 lines of statsmodels and itertools. For each subset size \(k\) we enumerate all \(\binom{p}{k}\) models, fit each with OLS, and record AIC, BIC, and adjusted \(R^2\). Then we pick the overall winner per criterion.
The simulated DGP loads on three of the five student predictors — study_hours_per_week, attendance_pct, and prior_GPA carry large weights, while sleep_hours_per_night and extracurricular_hours_per_week are smaller in magnitude and may not survive a strict BIC penalty. Which set of variables do you expect BIC to choose, and will the more permissive adjusted \(R^2\) pick the same set or a longer one?
Because the data are simulated from a process in which all five predictors carry non-zero coefficients but only three dominate the variance of the response, the BIC criterion typically selects the three large-effect predictors — study_hours_per_week, attendance_pct, and prior_GPA — while AIC and adjusted \(R^2\) often retain sleep_hours_per_night or extracurricular_hours_per_week as well. A single random draw of \(n = 300\) observations may occasionally include or exclude a marginal predictor when its sample correlation with the response happens to be inflated or attenuated. BIC penalises complexity more aggressively than AIC and therefore tends to favour the sparser model, while AIC is more permissive about including marginal predictors. When the two criteria agree, the result is unambiguous; when they disagree, the analyst should examine both and make a judgement informed by the substantive question.
Model Selection Criteria
Several criteria are available for choosing among candidate models, each with its own balance of fit and parsimony. Let \(SSE = \sum_{i=1}^n (y_i - \hat{y}_i)^2\) denote the sum of squared residuals and \(k\) the number of predictors in the candidate model.
| Criterion | Formula | Penalises | Select |
|---|---|---|---|
| Adj R\(^2\) | \(1 - \tfrac{(1-R^2)(n-1)}{n-k-1}\) | Weak | Maximise |
| AIC | \(n\ln(\tfrac{SSE}{n}) + 2k\) | Moderate | Minimise |
| BIC | \(n\ln(\tfrac{SSE}{n}) + k\ln n\) | Strong | Minimise |
| \(C_p\) | \(\tfrac{SSE}{\hat\sigma^2} - n + 2k\) | Moderate | \(C_p \approx k\) |
The criteria differ in the strength of their complexity penalties. Adjusted \(R^2\) applies the weakest penalty and is the most permissive about adding parameters. AIC and Mallow’s \(C_p\) apply moderate penalties calibrated for predictive accuracy. BIC applies the strongest penalty, growing logarithmically with the sample size, and is the most conservative about adding parameters. The practical guidance is that AIC and cross-validated metrics are appropriate when the analyst’s goal is prediction, while BIC is appropriate when the goal is to identify the true underlying model. For very small samples (typically \(n < 40\)), BIC’s stronger penalty tends to produce more robust selections; for large samples used in forecasting, AIC and cross-validation are preferred.
Understanding the Selection Criteria
Each criterion in the table combines a fit term — a function of the residual sum of squares that always improves as more predictors are added — with a penalty term that increases with the number of parameters. The column labelled “Penalises” indicates how aggressively the penalty pushes back against unwarranted complexity, ranging from the weak penalty of adjusted \(R^2\) to the strong penalty of BIC.
The selection rule for Mallow’s \(C_p\) takes the unusual form “select the model where \(C_p \approx k\)” because of a theoretical property: if the model is correctly specified, the expected value of \(C_p\) is approximately \(k\) itself. A \(C_p\) much larger than \(k\) indicates that important variables are missing and the model is biased; a \(C_p\) close to \(k\) indicates that the model is well specified; a \(C_p\) below \(k\) suggests possible overfitting in finite samples.
The statistical property that distinguishes BIC from AIC is consistency. As the sample size \(n\) tends to infinity, BIC selects the true model with probability approaching one, provided the true model is in the candidate set. AIC does not enjoy this guarantee. The reason lies in the penalty structures: BIC’s penalty \(k \ln n\) grows without bound as \(n\) increases, eventually overwhelming any spurious variable’s marginal contribution to fit, while AIC’s penalty \(2k\) remains fixed and cannot keep pace. The trade-off is fundamental: AIC is efficient, in the sense of producing the best out-of-sample predictions, while BIC is consistent, in the sense of identifying the true model structure. No criterion can be simultaneously efficient and consistent. For the practical problem of forecasting an individual student’s exam score the right choice is AIC, while for the inferential problem of identifying which habits genuinely drive performance the right choice is BIC.
Comparing 1-Feature, 3-Feature, and 5-Feature Student Models
The model comparison table presents the head-to-head results across the single-predictor specification, a parsimonious three-predictor specification that retains only the dominant habit variables, and the full five-predictor specification:
| Model | \(k\) | Train Adj R\(^2\) | Test R\(^2\) | BIC rank |
|---|---|---|---|---|
| study hours only | 1 | 0.40 | 0.38 | 3 |
| 3-feature | 3 | 0.84 | 0.83 | 1 |
| 5-feature | 5 | 0.85 | 0.84 | 2 |
The three-feature model — study hours, attendance, and prior GPA — wins on the BIC criterion. Its training adjusted \(R^2\) is essentially tied with the five-feature model, its test \(R^2\) is comparable, and its BIC ranking is best among the three because the BIC penalty grows quickly enough that the marginal contribution of sleep and extracurriculars does not justify the two additional parameters. The single-predictor model, by contrast, is too simple: it leaves systematic patterns in the residuals that the additional habit variables can explain, and its high bias reduces both adjusted \(R^2\) and test \(R^2\) relative to the larger alternatives. The data-driven conclusion is the three-feature model, an empirical confirmation of Occam’s razor: the right complexity is the smallest model that captures the genuine systematic structure, no more and no less.
Business Applications
The remaining sections apply the full multi-predictor regression machinery to two canonical decisions that arise in education analytics. The first is a forward-looking forecasting exercise: given a student’s habit profile, what is the model-implied expected final-exam score, and how does that benchmark prediction compare against the student’s own target? The second is a backward-looking inferential exercise: does a particular cohort, after controlling for its habit profile, exhibit a statistically detectable residual performance component — a positive or negative intercept shift — that would constitute evidence of either an unusually effective teaching intervention or an unaccounted-for source of variation? Both questions reuse the same fitted model, illustrating how a single regression supports multiple decisions across the life cycle of a course evaluation.
Application 1: Predict an Individual Student’s Exam Score
Scenario: A student studies 22 hours per week, attends 92 per cent of lectures, enters with a 3.5 GPA, sleeps 7 hours per night, and devotes 5 hours per week to extracurriculars. What does the model predict?
With a study-hours coefficient near 1.8 and a prior-GPA coefficient near 12, this habit profile is well above average on the two strongest predictors, and the model-implied score lands comfortably in the mid-90s.
This single prediction code block implements the daily workflow of an instructor or academic advisor compressed into three lines. First, the advisor observes the student’s habit profile — say, 22 hours of weekly study, 92 per cent attendance, a 3.5 GPA, 7 hours of sleep, and 5 hours of weekly extracurriculars. Second, those values are plugged into the fitted five-predictor model to obtain the point forecast for the final-exam score. Third, the forecast is compared with the student’s stated target. The resulting number is not itself an intervention. It is the benchmark prediction given the student’s habits — the score the student should earn on average given known behavioural patterns. A student who is targeting plus five points above the model-implied score is implicitly betting on positive idiosyncratic effort or test-day luck above what the habit variables predict, and the model makes that implicit gap explicit and measurable.
This workflow — habit-based expected score as baseline, with intervention targeting based on the gap between predicted and aspired performance — is the core logic of early-warning systems at large universities. The model does not need to predict absolute scores with perfect accuracy; it only needs to identify which students are most likely to fall below a passing threshold. That triage signal, aggregated across thousands of enrollments, allows advising resources to be concentrated where they have the largest expected impact.
Application 2: Test for a Significant Cohort Effect
Question: After controlling for habits, does this particular section of the course score systematically higher or lower than the model predicts?
A significantly non-zero intercept, after habits are controlled for, indicates a systematic cohort or course-level effect that the five habit predictors do not capture. In a well-specified model, the intercept absorbs the baseline level of the response and is large only because the regressors are not centred at zero.
The intercept test is the regression’s diagnostic counterpart to the slope tests, and unpacking its logic rewards care. In the simulated DGP the true intercept is exactly 35 points, so the typical estimate carries a tiny \(p\)-value and the analyst correctly rejects the null of zero intercept — but that rejection has no causal interpretation because the predictors are not centred at zero, and the intercept exists mostly to absorb the baseline level of exam scores. The more informative test would re-centre each predictor at its mean before fitting; then the intercept becomes the predicted score for an average student, and its standard error becomes the appropriate measure of how precisely the cohort’s mean performance is identified. A significant deviation of the centred intercept from a benchmark value would indicate that the cohort scored systematically above or below comparable cohorts — perhaps because the instructor introduced a new pedagogical intervention, or because the exam was easier than usual, or because the student population differed in characteristics not captured by the five habits. The alpha-style test on a cross-section has a direct analogue here: it asks whether the residual mean differs from the expected null after observable inputs have been controlled for.
Omitted variable bias in education models: Even the five-habit specification is not the “true” model. If a missing variable — instructor quality, prerequisite preparation, family income, or test anxiety — correlates with both the final-exam score and the included habits, then the estimated habit coefficients are biased. The Gauss-Markov theorem fails when relevant variables are omitted. Coefficient estimates from any habit model should be interpreted cautiously, with awareness that the “unexplained” residual variation may reflect missing background factors rather than genuine measurement noise.
Exercise: Analyse Your Own Cohort
Use a dataset of your own — a survey of classmates, the open student-performance datasets on the UCI Machine Learning Repository, or the simulated frame from the code above — to fit your own multiple-regression model.
- Choose a numeric response variable (a course grade, a percentile rank, or a standardised exam score).
- Identify between three and seven candidate predictors that you believe describe the response. Habits, prior preparation, demographics, and schedule load are all natural categories.
- Fit both a single-predictor model (using your best guess at the dominant driver) and the full multi-predictor model with
statsmodels. - Compare adjusted \(R^2\) across the two specifications. Do the extra predictors improve fit enough to justify the added parameters?
- Is the intercept statistically different from a reasonable benchmark value (for example, the average score in a previous cohort)?
- Which predictors are individually significant at the 5 per cent level, and which are not?
Hint: The five-predictor pipeline you built earlier in this section can be reused almost verbatim; only the data-loading line changes.
Chapter Wrap-up
Takeaway — The Regression Pipeline
The regression pipeline traces a six-stage workflow that applies to every applied study covered in this chapter, from CAPM to pharmacy profit prediction to the student-performance multiple regression. The analyst first explores the data with descriptive statistics and pairwise correlations to form initial hypotheses about which variables are related. The exploration stage feeds into visualisation: scatter plots and fitted-regression overlays that make patterns visible before any inferential machinery is applied. The build stage specifies the regression equation and fits it with sm.OLS. Evaluation compares fit and prediction error across train and test sets using \(R^2\) and RMSE. Selection narrows the candidate set of predictors using \(p\)-values, AIC, BIC, and adjusted \(R^2\). Application uses model.predict to generate point forecasts and get_prediction to attach the appropriate confidence and prediction intervals.
| Step | Tool |
|---|---|
| Explore | .corr() |
| Visualize | regplot |
| Build | sm.OLS |
| Evaluate | R\(^2\), RMSE |
| Select | p-values, AIC/BIC |
| Apply | .predict() |
The headline empirical lessons of this chapter follow directly from the workflow. Simple linear regression is the framework behind the Capital Asset Pricing Model: a single market factor captures the bulk of systematic risk, with beta as its scalar summary. Multiple linear regression generalises the same machinery to many predictors at once, as in the pharmacy-profit and student-performance examples, where five predictors collectively explain a substantially larger share of variation in the response than any single predictor alone. Variable selection insists that only predictors with genuine statistical support should be retained, lest noise be mistaken for signal. The next chapter shifts from supervised to unsupervised learning, replacing the predictive question of how \(y\) depends on \(x\) with the descriptive question of how observations group together based on their features alone — the territory of K-means and hierarchical clustering.
Chapter Summary
The Complete Regression Workflow
Every regression analysis follows the same five-stage pipeline. Skipping any stage is a professional error.
1. EXPLORE → scatter plots, correlations, descriptive stats
↓
2. MODEL → specify y = α + β₁x₁ + ... + βₖxₖ + ε
↓
3. ESTIMATE → OLS: minimize Σ(yᵢ - ŷᵢ)²
↓
4. INFER → t-tests, p-values, confidence intervals for each βⱼ
↓
5. DIAGNOSE → LINE residual plots: L (linearity), I (independence),
N (normality), E (equal variance)
↓
6. SELECT → Compare models: Adj-R², AIC, BIC, Cp
↓
7. PREDICT → point forecast ŷ*, CI for μ_Y, PI for individual YTop 10 Things to Check in Any Regression Output
When you receive a regression table — in an academic paper, a consultant’s report, or your own code — work through this checklist before trusting any number:
- F-statistic and overall p-value: Is the model significant at all? If \(p > 0.05\), stop — none of the variables may matter.
- Adjusted R²: How much variation does the model explain, penalized for complexity? Compare across models with different \(k\).
- Individual p-values: Which coefficients are statistically significant? Be cautious about over-reliance on a 0.05 threshold in the presence of multiple testing.
- Coefficient signs: Do they match economic intuition? A negative income effect on pharmacy profit would be a red flag requiring explanation.
- Coefficient magnitudes: Are they economically meaningful? A statistically significant but infinitesimally small coefficient may not matter in practice.
- Confidence intervals: Do any CIs contain zero? A CI that barely misses zero is less compelling than one centered far from zero.
- VIF for multicollinearity: Any VIF > 10 invalidates individual coefficient interpretation. Report and address.
- Residual plots (LINE): Did you check linearity, independence, normality, and equal variance? A regression without diagnostic plots is incomplete.
- Train vs. test performance: Is R² on held-out data close to in-sample R²? A large gap signals overfitting.
- Sample size and data quality: Were outliers examined? Is the data representative? Are there structural breaks (e.g., the COVID period changing all financial relationships)?
When NOT to Use Linear Regression
Linear regression is powerful but not universal. Recognize when to reach for a different tool:
- Binary outcome (\(y \in \{0,1\}\)): Use logistic regression. OLS can predict probabilities outside \([0,1]\), producing nonsensical results.
- Count outcome (\(y = 0, 1, 2, \ldots\)): Use Poisson regression. OLS may predict negative counts.
- Highly nonlinear relationships: If residual plots show clear curvature and polynomial terms don’t fix it, consider decision trees, gradient boosting (XGBoost), or neural networks. Linear regression imposes a global linear approximation; tree models adapt locally.
- Very high-dimensional predictors (\(p \gg n\)): Best subset selection is infeasible. Use LASSO, Ridge, or Elastic Net — regularized regression that handles hundreds or thousands of predictors.
- Time series with complex dynamics: Stock returns with GARCH volatility or mean reversion require GARCH/ARIMA models. OLS applied to serially correlated returns produces unreliable standard errors.
- Causal inference: If you need to estimate a causal effect (not just a predictive association), linear regression alone is insufficient. You need instrumental variables, difference-in-differences, regression discontinuity, or randomized assignment. The coefficient on an observational regressor is generally not a causal effect.
The most common misuse of regression in business: treating a statistically significant coefficient as evidence of a causal relationship. Income predicts pharmacy profit, but building a pharmacy in a high-income area will not cause income to rise. The coefficient reflects a correlational pattern, not a causal mechanism. Always ask: could there be a third variable driving both \(X\) and \(Y\)?
What’s Next: Chapter 4 — Clustering
Chapter 3 addressed supervised learning: we always had a target variable \(Y\) (NVDA return, pharmacy profit) and used regression to predict it from \(X\) features. The regression line is supervised by the labels.
Chapter 4 introduces unsupervised learning: there is no \(Y\). Instead, we discover hidden structure in \(X\) alone. K-means clustering partitions observations into groups based on proximity in feature space — no return to predict, no profit to explain, just “which observations are similar to each other?” The distance concept in K-means is the geometric cousin of OLS: both minimize a sum of squared distances, just in different directions.
The business applications shift accordingly: instead of predicting a customer’s lifetime value (regression), we ask “which customers are similar enough to segment together?” Market segmentation, portfolio construction by risk-return profile, and fraud detection all rely on clustering.
Further Reading
For deeper treatment of the topics in this chapter:
- Wooldridge, J.M. (2019). Introductory Econometrics: A Modern Approach (7th ed.). The standard undergraduate econometrics textbook — thorough on inference, panel data, and time series.
- Greene, W.H. (2018). Econometric Analysis (8th ed.). The graduate-level reference — covers GLS, IV, panel models, and limited dependent variables rigorously.
- Stock, J.H. and Watson, M.W. (2019). Introduction to Econometrics (4th ed.). Accessible introduction with strong emphasis on causal inference and policy applications.
- Fama, E.F. and French, K.R. (2015). “A five-factor asset pricing model.” Journal of Financial Economics, 116(1), 1-22. The original five-factor paper.
- Harvey, C.R., Liu, Y., and Zhu, H. (2016). “… and the Cross-Section of Expected Returns.” Review of Financial Studies, 29(1), 5-68. The multiple testing problem in factor research.
- Pearl, J. and Mackenzie, D. (2018). The Book of Why: The New Science of Cause and Effect. Accessible introduction to causal inference and the difference between correlation and causation.
- Hastie, T., Tibshirani, R., and Friedman, J. (2009). The Elements of Statistical Learning (2nd ed.). The bridge between classical regression and modern machine learning — covers LASSO, ridge regression, and variable selection comprehensively.
Debug Yourself: The Missing Constant
The cell below tries to recover a known intercept of 0.5 and a known slope of 2.0 from simulated data. It runs without raising an exception, but the hidden test at the end fails because the printed intercept is nowhere near 0.5. There is a single-line bug. Try to find it before reading the explanation that follows.
The trap is that sm.OLS(y, x) does not add an intercept on its own — unlike R’s lm() or Stata’s regress, which always include one by default, statsmodels fits whatever design matrix you hand it. With only the raw x column passed in, the model is forced through the origin and model.params contains a single number that you then misread as the intercept. The fix is to build the design matrix explicitly: X = sm.add_constant(x) and then sm.OLS(y, X).fit(). This is the most common source of suspicious-looking regression output for anyone arriving from another language, and it surfaces every term in this course.
Interactive Explorer: How β and Noise Change a Regression
The two values at the top of the cell — beta (the true slope) and noise_sigma (the standard deviation of the idiosyncratic shock) — control how cleanly OLS can recover the underlying relationship. Edit them, press Run, and watch how the fitted line, the 95 % confidence interval for \(\beta\), \(R^2\), and the RMSE all change. Larger beta tilts the line more steeply but does not by itself widen the confidence interval; larger noise_sigma leaves the slope unchanged on average but inflates the confidence interval, drives \(R^2\) down, and pushes RMSE up. Try beta = 1.0 with noise_sigma = 0.005 for a textbook market-tracking stock, then beta = 2.0 with noise_sigma = 0.05 for an aggressive name whose idiosyncratic noise drowns much of the systematic signal.
Working with an AI Copilot
A common workflow in 2026 is to paste a model.summary() table into an AI assistant and ask “is the coefficient on \(X\) significant?” The assistant will read the \(p\)-value column and answer, often correctly. The problem is what it does not say. It will not flag that the standard errors were inflated by heteroskedasticity, that the residuals are autocorrelated, or that an \(R^2\) of 0.99 is suspicious because a near-duplicate of the response variable was accidentally included as a predictor. The AI reads the numbers in the table; it does not read what those numbers mean about the data-generating process.
Three rules turn the copilot into a useful collaborator. First, for every “is this significant?” question, also ask the AI to walk through whether the LINE assumptions — Linearity, Independence, Normality of residuals, and Equal variance — appear to hold. Second, follow up with “what would change this conclusion?” — robust standard errors, dropping influential observations, and out-of-sample tests matter more than a single \(p\)-value. Third, recognise that the AI cannot detect a data leak. That requires human judgment about how the data was collected and what the predictor could plausibly know at prediction time.
Decision Memo — Should the Team Sign This Player?
A regression table is not a decision. To convert estimates into action, analysts write a short decision memo that tells a non-technical reader what to do, why, and what could go wrong. The format below is the one used by most consulting firms and front offices: recommendation first, evidence second, caveats third, and a concrete next step at the bottom. Practise compressing your regression output into this structure — it is the deliverable your future managers will actually read.
To: General Manager, Eastern Conference team From:
, basketball analytics intern Subject: Sign free agent X to a 1-year prove-it deal Date: 2026-05-15 Recommendation: Sign player X on a 1-year, performance-incentivised contract.
Evidence: - Regression on last 82-game season: every +1 assist → +0.04 win probability (p < 0.01). - Player X averaged 6.2 assists/game last year, vs. league median 3.8. - Turnover rate is league-average (β = −0.05, p = 0.03 in our model).
Caveats: - Linear probability model: predicted win probabilities outside [0, 1] are common. - Assists are partially endogenous — better teammates create more assists. - 12-row sample is illustrative only; real decision needs 200+ games.
Next step: Re-run with logistic regression and full 82-game season; build hold-out test on next 20 games.
Long-Term Capital Management was founded in 1994 by veterans of Salomon Brothers and quickly recruited two of the most decorated theorists in finance: Robert Merton and Myron Scholes, who shared the 1997 Nobel Memorial Prize in Economics for the option-pricing framework that bears their names. By 1998 the fund had grown to roughly $4.8 billion in equity supporting more than $129 billion in positions, plus an off-balance-sheet derivatives book in the trillions. Its core business was fixed-income arbitrage: small mispricings between similar bonds, leveraged aggressively. The risk models assumed that credit spreads were drawn from an approximately normal distribution estimated on the prior decade of data.
In August 1998, Russia defaulted on its rouble-denominated debt and devalued. Credit spreads on the trades LTCM held moved by more than eight standard deviations in a single week — an event whose probability under the fund’s normal-distribution assumption was roughly 1 in \(10^{20}\), less than the chance of a single specified atom being chosen at random from all the atoms on Earth. LTCM lost $4.6 billion in four months and had to be rescued in a $3.6 billion bailout coordinated by the Federal Reserve Bank of New York and fourteen Wall Street banks.
The lesson for regression is sharper than “use fatter tails.” Every linear-regression result is conditional on the distributional assumptions you fed in. When you fit a model on 2010–2020 data and predict 2030, you are implicitly betting that the next decade is drawn from the same distribution. LTCM’s coefficient estimates were not wrong; their world model was. The financial crisis happens in the residuals.
Review Cards — Chapter 3
These cards are scheduled for spaced repetition. Click Show answer, rate how easily you recalled it, and the book will surface the harder cards again on a longer interval — the same Anki technique used to study language and medical material. Your review history is stored privately in your browser; nothing is uploaded.
About 55 % (\(r^2 = 0.74^2 \approx 0.55\)), not 74 %. A common mistake is to interpret \(r\) itself as the share of explained variation — that’s \(r^2\). The remaining 45 % is the variance of the residuals, which is exactly what diversification or additional predictors might be able to reach.
\(r_{\text{stock}} - r_f = \alpha + \beta\,(r_m - r_f) + \epsilon\). Both sides are excess over the risk-free rate. The intercept \(\alpha\) is Jensen’s alpha — return above what market exposure predicts — and the slope \(\beta\) is the stock’s sensitivity to the market factor.
\(\beta > 1\) is an aggressive stock: it amplifies market moves in both directions, so a 1 % move in the market produces a more-than-1 % expected move in the stock. \(\beta < 1\) is a defensive stock: it dampens market moves. \(\beta = 1\) corresponds to a stock that, on average, moves one-for-one with the market.
Jensen’s \(\alpha\) is the intercept in the excess-return CAPM regression — the average daily return earned above what the stock’s market exposure predicts. In an efficient market, any persistent positive \(\alpha\) attracts arbitrage capital that quickly bids prices up and drives the alpha back toward zero. For most real stocks, \(\hat{\alpha}\) is small in magnitude and not statistically distinguishable from zero.
coef is the point estimate \(\hat{\beta}_j\). std err is \(SE(\hat{\beta}_j)\). t is the ratio \(\hat{\beta}_j / SE(\hat{\beta}_j)\), the test statistic for \(H_0: \beta_j = 0\). P>|t| is the two-tailed \(p\)-value for that test. The bracket is the 95 % confidence interval \(\hat{\beta}_j \pm t_{0.025,\,n-k-1}\cdot SE(\hat{\beta}_j)\). A useful equivalence: the interval contains zero iff \(p > 0.05\).
The CI for \(\mu_Y\) captures uncertainty about the average response at a given \(x^*\); it shrinks as \(n\) grows. The PI for \(Y\) captures uncertainty about one specific future observation at \(x^*\); it includes the irreducible noise term \(\epsilon\), so it is always wider and never collapses to a line even as \(n \to \infty\). Risk managers care about the PI (single trading day); portfolio benchmarking cares about the CI (average across many days).
False. An insignificant overall F-test means the data do not provide strong evidence that at least one slope is non-zero in this sample. It does not prove that every slope is exactly zero. A single genuinely useful predictor can be masked by noise, by multicollinearity, or by an underpowered sample. Read an insignificant F as “the evidence is weak here”, not “the variables are useless.”
Because shuffling lets future observations leak into the training set — a textbook case of look-ahead bias. The resulting test performance is artificially inflated and the model has effectively been allowed to peek at the answers. For stock returns, always sort by date, train on earlier dates, and evaluate on later dates. The same rule applies to any panel with a temporal dependence structure.
Because statsmodels does not add an intercept by default — unlike R’s lm() or Stata’s regress. With only the raw x column passed in, the regression is forced through the origin and model.params contains a single slope, no intercept. The fix is X = sm.add_constant(x) followed by sm.OLS(y, X).fit(). Forgetting this is the single most common bug for analysts coming from R or Stata.