lecture-01-the-geometry-of-linear-equations

18.06 Lecture 1: The Geometry of Linear Equations

Strang’s opening lecture. Sets up the row picture vs column picture for systems of linear equations and quietly plants the seeds for most of the course.

The setup

Given a system like:

2x - y      = 0
-x + 2y - z = -1
    -3y + 4z = 4

You can write this as Ax = b where:

• A is the matrix of coefficients
• x is the column vector of unknowns [x, y, z]
• b is the column vector of right-hand sides [0, -1, 4]

The same nine numbers in A carry two different geometric stories depending on whether you read them by row or by column. Both stories are valid. They give the same answer. They’re both worth seeing.

The row picture

Each row of A is the coefficients of one equation. Each equation defines a plane in 3D (or a line in 2D). The solution to the system is the point where all the planes intersect.

This is the picture you grew up with from high school algebra: “find the point where these equations all agree.”

Why it’s limited: works fine for visualizing 2 or 3 equations, but completely breaks as a mental image past that. You can’t really picture four hyperplanes intersecting in 4D. The row picture doesn’t scale.

The column picture

Each column of A is treated as a vector — an arrow in 3D space. The equation Ax = b is asking:

What recipe of these column vectors — how much of column 1, how much of column 2, how much of column 3 — gets me to the target vector b?

The unknowns x, y, z are the weights in the recipe.

This reframes “solve a system of equations” as “find the right combination of arrows.” Same problem, completely different mental model.

Why both pictures give the same answer

This struck me as suspicious at first — we’re using the same nine numbers in two completely different geometric roles. Why does that work?

It’s not coincidence. Matrix-vector multiplication has two equivalent definitions:

Row way: the i-th entry of Ax is (row i) · x (dot product).

Column way: Ax = x₁·(col 1) + x₂·(col 2) + x₃·(col 3) (linear combination of columns weighted by entries of x).

These produce literally the same numbers. It falls out of the distributive property in a few lines. Matrix multiplication is defined in exactly the way that makes these two interpretations agree. If it were defined any other way, the row and column pictures would disagree and the whole framework would collapse.

So both stories are baked into the same arithmetic. You get to pick whichever one is more useful for the question you’re asking.

The big question: can Ax = b always be solved?

In the column picture this becomes: can you reach any target b by combining the columns?

The answer depends on whether the columns provide enough genuinely different directions.

Cases (in 3D, with 3 columns)

The reachable region is called the column space. Its dimension (3, 2, 1, or 0) is called the rank. Strang doesn’t use those words yet — he’s saving them — but the question “can Ax = b always be solved?” is exactly the question those words are built to answer.

Subtle point: pairwise vs. collective dependence

My first instinct was that the system fails when “two columns point the same direction.” That’s true but it’s only the simplest case.

The real condition is linear dependence: one of the columns can be written as a combination of the others. Three arrows in 3D can be pairwise non-parallel (no two collinear) but still all lie in the same plane — and then collectively they only span 2D.

The technical statement: vectors v₁, ..., vₙ are linearly dependent if there exist coefficients c₁, ..., cₙ (not all zero) such that:

c₁v₁ + c₂v₂ + ... + cₙvₙ = 0

That equation is the cancellation. The vectors aren’t pointing the same way individually, but there’s a non-trivial way to weight them that makes them collectively annihilate.

Higher dimensions create new ways to be dependent

• In 2D: only collinearity causes dependence.
• In 3D: coplanarity is a new failure mode (3 vectors, no two parallel, but all in one plane).
• In nD: you can have many vectors that all look pairwise distinct but collectively lie in some lower-dimensional subspace.

This is exactly the manifold story for embeddings. A 768D embedding model rarely uses all 768 dimensions of structure — the meaningful content typically lives on a much lower-dimensional manifold inside the ambient 768D space. The “extra” dimensions are slack created by approximate linear dependence in the data.

Independence and uniqueness

When the columns are linearly independent:

• Every reachable b has exactly one recipe.
• Each variable in x is doing irreplaceable work — no coefficient can be substituted by a combination of the others.

When the columns are dependent:

• Recipes are no longer unique. If c₁v₁ + c₂v₂ + c₃v₃ = 0 is a non-trivial cancellation, then for any solution x, the vector x + (c₁, c₂, c₃) is also a solution. Adding zero doesn’t change the result.
• The system either has no solution or infinitely many — never exactly one.

Independent columns that span the column space form a basis. A basis is the Goldilocks condition: enough vectors to span, not so many that recipes become non-unique.

Connections to actual work

Embeddings

A 768D embedding is a vector in 768D space. Cosine similarity between embeddings is asking a column-picture question: do these vectors point in similar directions?

In high-dimensional space, two random unit vectors are almost certainly nearly orthogonal. So when two embeddings do have high cosine similarity, that’s earned signal — the model placed them near each other against the overwhelming statistical pressure for everything to be perpendicular.

Multicollinearity in regression

When predictive features are linearly dependent (or near-dependent), regression coefficients aren’t uniquely determined. You can’t interpret which feature is doing the work, because the work can be redistributed among the dependent features without changing the model’s output. Same phenomenon as non-unique recipes, dressed in stats vocabulary.

Graphics

The pygame cube exercise is doing this whole picture. Each rotation matrix is a linear transformation. Composing rotations = multiplying matrices. The 3D-to-2D projection is literally applying a [[1,0,0],[0,1,0]] projection matrix (just inlined). Translation isn’t linear, which is why graphics pipelines use homogeneous coordinates (4x4 matrices in 4D) — that’s how you fold translation into matrix multiplication.

Looking ahead

The four fundamental subspaces (column space, null space, row space, left null space) are coming. The questions sketched above — what can be reached, what cancels to zero, what dimension is the reachable region — are the questions those subspaces formalize. Strang will spend much of the course on them because they’re the skeleton of everything else.

Eigendecomposition and SVD (later in the course) are answers to a deeper question: can we find a coordinate system in which a matrix’s action becomes simple — diagonal, each variable scaled independently with no cross-talk? Often yes. SVD says: always yes, for any matrix.

Open questions / things to revisit

• The mechanics of homogeneous coordinates — worth coming back to once vector spaces are formalized.
• The exact statement and proof that row rank = column rank — Strang will get there. The “huh, why do both interpretations of the matrix work?” feeling from lecture 1 is the seed of this theorem.
• Refactor the pygame cube to (a) precompute the composed rotation matrix once per frame and (b) stack vertices as columns of a single 3×8 matrix so the whole rotation is one matrix multiply.