---
title: "Overfitting and regularisation"
url: https://manishshivanandhan.com/notes/overfitting-and-regularisation/
description: "A model that memorises its training set looks excellent right up to the moment it meets new data."
published: 2026-05-21
updated: 2026-05-21
---

# Overfitting and regularisation

Published 2026-05-21

Machine learning workflow · applies to: Model capacity, training diagnostics · last updated 2016-06-04

A model that memorises its training set looks excellent right up to the moment it meets new data.

## What the curves look like

Plot the loss on the training split and on a held-out split against training progress. Early on both fall together. If capacity exceeds what the data supports, the training curve keeps falling while the held-out curve flattens and then turns upward. The gap between them is the memorisation.

Two curves that are both high and flat mean the opposite problem: the model has too little capacity, or the columns simply do not carry the signal being asked for.

## The usual remedies

More data is the remedy that always works and is usually unavailable. After that: reduce capacity, penalise large parameters, stop training at the point the held-out curve turned, or randomly disable parts of the network during training so no single path can carry the whole answer.

A penalty on parameter size pushes the model towards simpler explanations. Applying it as a sum of squares shrinks everything gently; applying it as a sum of absolute values drives some parameters to exactly zero and quietly performs feature selection.

## Keeping the held-out split honest

The held-out split stops being a fair estimate the moment decisions start being made from it. Choosing a stopping point, a capacity or a penalty strength by looking at it means it has become part of training. Anything reported as a final number needs a third split that no decision has touched.

## Diagnosing a gap

1. Plot training and held-out loss on the same axes against the same progress unit.
2. Find the point where the held-out curve stops improving.
3. If the training curve is still falling past that point, the model has spare capacity to memorise.
4. Add a penalty or reduce capacity, retrain, and compare the gap rather than the absolute numbers.
5. Report the final figure from a split that took part in no decision.

```
sum of squares:          cost = loss + lam * sum(w ** 2)
sum of absolute values:  cost = loss + lam * sum(abs(w))
```

The two penalty shapes

Worth knowing

If the held-out curve is noisier than the effect being measured, the held-out split is too small to support the decision being made from it.

[Previous: Reading a confusion matrix](https://manishshivanandhan.com/notes/reading-a-confusion-matrix/)

[Next: Force directed graphs in D3](https://manishshivanandhan.com/notes/force-directed-graphs-in-d3/)

[Return to Machine learning workflow](https://manishshivanandhan.com/topics/machine-learning/)