FiledSeptember 16, 2026·6 min read·By The Editor

L1 vs L2 Regularization

Both charge a model a fee for large weights. L1 uses a diamond and gives you weights of exactly zero; L2 uses a circle and gives you weights that are merely small. One picture explains the whole difference.

regularizationtheoryintuition

Both do the same job — stop a model memorising its training data — and they do it the same way: by charging the model a fee for large weights. The difference is what the fee is calculated on, and that one detail changes what you end up with.

L1 hands you a model with fewer features. L2 hands you a model with smaller ones. Here is why.

The setup

A model trains by minimising its loss — how wrong it is on the training data. Left alone it will happily drive that to near zero by memorising quirks that will never appear again. See the bias-variance bargain for what that costs you.

Regularization adds a second term. Now the model minimises:

\text{total} = \underbrace{\text{how wrong I am}}_{\text{loss}} + \lambda \cdot \underbrace{\text{how big my weights are}}_{\text{penalty}}

The only question is how you measure "how big my weights are".

\text{L1} = \lambda \sum_i |w_i| \qquad\qquad \text{L2} = \lambda \sum_i w_i^2

In plain English

L1 adds up the weights ignoring their signs. L2 adds up their squares. λ (lambda) is the fee rate — turn it up and the model cares more about staying small than about being right. That is the whole difference in notation. What follows is why it matters.

Why L1 zeroes weights and L2 does not

Figure 1 · why one zeroes weights and the other does not

This picture is the difference. Both penalties say “keep the weights inside this shape.” The best model is wherever the closing loss contours first touch it. A diamond gets touched at a corner — and corners sit on the axes, where a weight is exactly zero. A circle has no corners, so it gets touched mid-edge, where both weights are small but alive.

Think of the penalty as a budget. "Keep your weights inside this shape." L1's budget is a diamond; L2's is a circle. The best model is wherever the loss contours — rings of equally-good fits, drifting in from the unpenalised answer — first touch the budget shape.

A diamond has corners, and its corners sit on the axes. Landing on an axis means one weight is exactly, precisely zero. Contours drifting in from an angle hit a corner far more often than they hit a flat edge, so L1 keeps producing answers with genuine zeros in them.

A circle has no corners. Every point on it is a smooth curve, and the touch point almost never lands exactly on an axis. So L2 gives you weights that are small — 0.03, 0.007 — but essentially never 0.

The same thing in one dimension

Figure 2 · the same story in one dimension

The V has a constant slope, so the reward for shrinking a weight is the same whether it is large or nearly gone — exactly the pressure needed to push it the last step to zero. The parabola flattens near zero: once a weight is small, L2 stops caring.

Look at what each penalty charges as a weight approaches zero. L1 is a V: its slope is 1 the whole way down. Shrinking a weight from 0.1 to 0 pays exactly as well as shrinking from 10 to 9.9 — so there is always pressure to take the last step.

L2 is a parabola, and near zero it flattens out. Once a weight is small, shrinking it further buys almost nothing, so gradient descent leaves it alone. L2 has no incentive to finish the job.

In plain English

L1 pays a flat bounty per unit of shrinkage, so it pushes small weights all the way to zero. L2 pays a bounty proportional to size, so it crushes big weights hard and then loses interest. That is the entire mechanism.

What that means in practice

L1 performs feature selection. Feed it 500 columns and it will hand back a model using 40 of them, with 460 weights sitting at exactly zero. You can delete those columns. The model has told you they were not earning their place.

L2 performs shrinkage. Feed it the same 500 columns and it uses all 500, each with a modest weight. Nothing is discarded; everything is damped.

This leads to the one failure mode worth knowing: give L1 two nearly-identical features and it picks one arbitrarily and zeroes the other. Height in centimetres and height in inches — L1 keeps whichever it happened to like, discards the other, and its choice can flip if you retrain on slightly different data. L2 splits the credit between them and is stable. If your features are correlated and you care about which ones the model says matter, that instability is a real problem.

Use L1 when

You have far more features than you can justify, and you want the model to tell you which ones to throw away.

Use L2 when

You want every feature to contribute a little, and you want smooth, stable training. This is the default, and it is the right default.

Use both

Elastic net. Groups of correlated features get shared credit instead of one winner taking all.

Choosing lambda

Neither penalty means anything without λ, and λ is not something you derive — it is something you search for.

λ too small — the penalty is ignored, the model overfits, and you have gained nothing.
λ too large — the penalty dominates, every weight is crushed toward zero, and the model underfits everything equally.
λ about right — found by trying a range on a validation set, not by reasoning.

The usual approach is a logarithmic sweep — 0.0001, 0.001, 0.01, 0.1, 1 — scored on held-out data. The shape of that curve is almost always a U: too little regularization is bad, too much is bad, and there is a basin in the middle that is wide enough to hit.

One thing that trips people up

Scale your features first. Both penalties charge per unit of weight, and a feature measured in millimetres needs a weight a thousand times larger than the same feature in metres to say the same thing. Without standardising, the penalty quietly punishes features for their units rather than their usefulness.

The short version

Both add a fee for large weights. λ sets the fee rate.
L1 sums absolute values. Its budget shape has corners on the axes, and its penalty has constant slope. Result: weights hit exactly zero, and you get feature selection.
L2 sums squares. Its budget shape is smooth, and its penalty flattens near zero. Result: weights get small but survive, and you get stable shrinkage.
Correlated features: L1 picks one at random, L2 shares the load.
Standardise first, then sweep λ on a validation set.

Overfitting vs underfitting — the problem these penalties exist to solve.
Hyperparameters vs parameters — how to search for λ.
The bias-variance bargain — why penalising a model for being clever makes it better.
Gradient descent, by hand — the process that both penalties are quietly steering.

← Archive— end of essay —Front page →