AIC and BIC in Regression: How Penalized Likelihood Scores Pick Models

AIC and BIC in regression are penalized likelihood scores that rank candidate models by fit minus a complexity penalty. AIC subtracts 2k from twice the maximized log-likelihood; BIC subtracts k times the natural log of n from the same quantity. Lower values win. AIC targets the model that best predicts new data and tends to overfit slightly as n grows. BIC targets the true model if it sits inside the candidate set, and it punishes extra parameters harder once n exceeds about 8 observations. Neither score is a hypothesis test, and neither can rescue a misspecified model.

What the Two Scores Actually Compute

Both formulas start from the same quantity: the maximized log-likelihood, written L-hat. For a Gaussian linear regression with p predictors and n rows, L-hat is maximized by ordinary least squares, so the likelihood term reduces to n times ln(RSS/n), where RSS is the residual sum of squares. The two criteria then differ only in penalty.

The AIC formula is AIC = 2k − 2L-hat, where k counts every estimated parameter including the intercept and the error variance. The BIC formula is BIC = k·ln(n) − 2L-hat.

That single substitution, 2 for ln(n), is the entire debate. At n = 8, ln(8) is 2.08, so the criteria nearly tie. At n = 1,000, ln(1,000) is 6.91, so BIC charges each parameter 3.45 times more than AIC does.

In plain terms, AIC charges a flat fee per variable; BIC raises the fee as your sample grows.

The absolute values are meaningless. An AIC of 4,812 is not good or bad on its own. Scores are only comparable across models fit to the same rows with the same response transformation. Refit on a log scale and the constant shifts by the sum of the logs of the response.

Worked example: a regression with n = 500, k = 6, and RSS = 1,200. Then n·ln(RSS/n) = 500 × ln(2.4) = 500 × 0.8755 = 437.75. Twice that is −875.5 in the −2L-hat form, so AIC = 12 + 875.5 = 887.5 and BIC = 6 × 6.215 + 875.5 = 912.8. Drop a predictor and let RSS rise to 1,260: AIC falls to 885.2 but BIC falls to 905.8. The extra variable survives under AIC and dies under BIC. That divergence is typical, not pathological.

Why the Penalty Exists at All

Adding parameters never increases RSS, so raw fit always favors the biggest model. The penalty corrects for that mechanical optimism. Each free parameter buys roughly 2 units of expected in-sample log-likelihood gain that will not replicate out of sample; AIC subtracts exactly that expectation, which is why it is an unbiased estimate of predictive divergence up to a constant.

AIC estimates the Kullback-Leibler divergence between the fitted model and the unknown truth. Choosing minimum AIC is asymptotically equivalent to choosing the model with the smallest expected predictive loss, but only among the candidates you supplied. It is not estimating the probability that a model is correct. This is the most common professional caveat: AIC ranks, it does not endorse.

BIC comes from a Bayesian approximation of the posterior probability of each model under equal priors. Its heavier penalty follows from the Laplace approximation of the marginal likelihood, where the ln(n) term appears naturally as sample size grows.

AIC vs BIC: Head-to-Head Numbers

The table below reconciles the practical thresholds. A difference of 2 in AIC is weak evidence; 4 to 7 is considerable; above 10 the lower score is strongly preferred. For BIC, differences of 2 to 6 are positive, 6 to 10 strong, and above 10 very strong, per Kass and Raftery (1995). Note that AIC and BIC differences cannot be compared to each other's thresholds, because the penalty scales differ.

The AICc correction matters more than most tutorials admit. When n = 30 and k = 8, the correction adds 2×8×9/(30−8−1) = 6.86 units, larger than most genuine AIC differences. Use AICc whenever n/k is below 40, per the recommendation popularized by Burnham and Anderson (2002).

One unit system note: all penalties are in natural-log units, and switching to base-10 logs would rescale every threshold by 2.303 without changing any ranking. Do not mix conventions across software packages.

What AIC and BIC Cannot Do

Neither score detects a missing variable that no candidate captures. If the true driver is absent from the set, both criteria will confidently rank the best of a bad lot. This is the classic misspecification blind spot: AIC and BIC compare, they do not search.

Neither score tests a null hypothesis. A likelihood ratio test needs nested models and a chi-square reference distribution. AIC and BIC apply to non-nested models and are descriptive.

Neither score handles collinearity gracefully. Two near-identical predictors can split their explanatory power, and the penalty counts both parameters. Some analysts see the score punish a pair that jointly deserves one slot.

Neither score validates the assumption of independent errors. With autocorrelated residuals, the effective sample size is smaller than n, so BIC's ln(n) penalty is too harsh and AIC's 2 is too lenient. Both mislead in the same direction, toward models that look better than they are.

Red flags that your comparison is invalid:

• Models fit on different rows, different response scales, or different transformations.
• Comparing a model with the intercept removed to one with it retained, since k changes by one silently.
• Reporting a single winning score without the runner-up's value or the raw difference.
• Using BIC with n under 50 and treating the result as decisive.
• Treating a score gap of 1.5 as meaningful when sampling noise is larger.
• Quoting AIC thresholds against a BIC difference.

Small Samples, Big Parameters, and Other Edge Cases

The n/k ratio is the governing number. The table below shows how the evidence threshold translates into real decision weight across sample sizes.

The counterintuitive result sits in the last two rows. At very large n, BIC can underfit relative to predictive goals. A real but small effect, say a coefficient explaining 0.3 percent of variance, will fail the ln(n) hurdle once n reaches tens of thousands while genuinely improving forecasts. If your goal is prediction at scale, AIC or cross-validation should outrank BIC even though BIC is the "stricter" criterion.

Boundary effects are another edge case. Linear regression with p close to n produces unstable L-hat and inflated differences. Ridge or lasso penalties replace the discrete k with an effective degrees of freedom, and the AIC-style penalty must be recomputed as the trace of the smoother matrix, not the raw parameter count. Most consumer software hides this; check that your package reports effective degrees of freedom.

Common Misconceptions, Corrected

The biggest misconception is that the lower AIC model is "the true model." It is the estimated best approximation among candidates. A competitor 3 units higher has roughly a 1-in-5 likelihood of actually being better, not a 1-in-1,000 chance, because AIC differences convert to relative likelihood via exp(−Δ/2).

A second misconception treats BIC as universally superior because it is consistent. Consistency is an asymptotic property; at finite n it describes a limit, not a guarantee. Simulations at n = 100 to 200 routinely show AIC selecting the better predictor while BIC drops a modest real effect.

A third misconception assumes the scores need different data. They need the same rows and the same response. Many apparent contradictions trace to packages defaulting to different error-variance conventions or omitting the intercept from k.

A fourth: an AIC difference of 2 is often called "significant." It corresponds to a likelihood ratio of about 2.7, which is suggestive, not conclusive. A difference of 10 corresponds to roughly 148:1 and is the point where practitioners stop arguing.

Practical Actions: How to Run the Comparison

Follow this sequence, in this order, and the scores will behave.

• Fix the row set, the response scale, and the intercept convention before fitting anything.
• Fit every candidate on identical data and record k explicitly, including the variance parameter.
• Compute AIC, AICc, and BIC together; report all three plus the delta to the best model.
• Confirm the winner with 10-fold cross-validation; if CV disagrees sharply, trust CV for prediction tasks.
• Check residual diagnostics before trusting any score; a violated error assumption invalidates all three.
• Report the score gap, not just the winner, so readers can see how close the call was.

In plain terms: use AIC and AICc when you want to predict, use BIC when you want to explain and you have thousands of rows, and always show your deltas.

For any number that will be published as a model claim, treat a gap under 2 as a tie. Report ties as ties. The scores are decision aids, not verdicts, and the most robust published analyses show the runner-up's value alongside the winner's.

Where These Criteria Fit Against Modern Alternatives

AIC and BIC remain the default in classical regression because they are cheap and interpretable. Cross-validation buys better predictive honesty at 5 to 10 times the compute, and it does not require n/k assumptions. Regularized regression with a tuned penalty embeds the complexity question inside the fit rather than after it.

The honest position is that the two scores answer slightly different questions, and choosing between them is a statement about your goal. A model selection workflow that names AIC or BIC in its methods section and reports the runner-up has done what the literature asks. One that reports a single number has not.