Determinants

The determinant is a scalar value that encodes both the volume and orientation of the geometric object (parallelepiped) formed by the rows or columns of a matrix.

Determinant in Two Dimensions (Area)

Suppose we have two vectors in :

If we put these as rows (or columns) of a matrix:

then the determinant is:

Geometric Interpretation

  • The parallelogram spanned by and has area
  • The sign of indicates orientation:
    • Positive: if rotated counterclockwise to
    • Negative: if clockwise

So in 2D, the determinant is the signed area of the parallelogram formed by the two vectors.

Determinant in Three Dimensions (Volume)

Consider three vectors in :

If we arrange them as rows of a matrix , then:

  • gives the volume of the parallelepiped formed by

General Case in

For an matrix:

  • gives the -dimensional volume (hypervolume) of the parallelepiped spanned by the rows (or columns)
  • The sign encodes orientation relative to the standard basis

For any square matrix :


Matrix Notation for Dot Product

The matrix notation is used because the mechanical process of multiplying the matrices and results in the exact same calculation as the dot product .

Suppose we have the column vectors:

1. The Dot Product ()

The standard dot product is calculated by multiplying corresponding components and adding them up:

2. The Matrix Notation ()

First, we find the transpose of , which turns the column vector into a row vector:

Now, we perform standard matrix multiplication of (a matrix) and (a matrix):

The rule for matrix multiplication is “row times column”:

The result is a matrix, which is just a single number (a scalar).


Triangular Matrices and Their Determinants

Lower Triangular Matrix

A lower triangular matrix is a square matrix where all the entries above the main diagonal are zero.

Example of a 4×4 lower triangular matrix:

Upper Triangular Matrix

An upper triangular matrix is the opposite, with all entries below the main diagonal being zero.

Example of a 4×4 upper triangular matrix:

Determinant of a Triangular Matrix

For both upper and lower triangular matrices, the determinant is simply the product of the entries on the main diagonal.

Block Upper Triangular Matrices

Recognizing a block triangular structure is a major computational shortcut, especially for finding determinants and eigenvalues.

Simplifying Determinants

The determinant of a block triangular matrix is simply the product of the determinants of its diagonal blocks.

For example:

Instead of computing a difficult 4×4 determinant, you only need to calculate two simple 2×2 determinants:

The determinant of the full matrix is just their product:

Finding Eigenvalues

The set of eigenvalues of a block triangular matrix is the union of the eigenvalues of its diagonal blocks.

This means you can:

  1. Find eigenvalues of → get
  2. Find eigenvalues of → get
  3. Eigenvalues of =

A matrix can be partitioned into any number of blocks of any size, as long as the partitioning is consistent.

Applications

  1. Markov Chains: States often group into communicating classes → block structure
  2. Network Analysis: Communities in networks lead to block patterns
  3. Differential Equations: Decoupled or weakly coupled systems
  4. Quantum Mechanics: Block diagonal Hamiltonians for non-interacting subsystems

Row Operations and Determinants

Row operations preserve the linear dependency relationships between the columns.

1. Row Swap

Swapping two rows multiplies the determinant by .

2. Row Addition

Adding a multiple of one row to another does not change the determinant.

3. Row Scaling

Multiplying a row by a non-zero scalar multiplies the determinant by .

Example

Let’s compute the determinant of:

Row Operations to Create Upper Triangular Form

Operation 1: (no change to determinant)

Operation 2: (no change to determinant)

Operation 3: (no change to determinant)

For an upper triangular matrix, the determinant is the product of diagonal elements:


Elementary Matrices

An elementary matrix is a square matrix obtained from the identity matrix by performing exactly one elementary row operation. Pre-multiplying any matrix by an elementary matrix applies that same row operation to the matrix as a whole.

  • If is elementary and is any matrix, then is with one row operation applied.
  • Right-multiplication by an elementary matrix applies the corresponding column operation: .

Laplace (Cofactor) Expansion for Determinants

For an matrix , the Laplace expansion along row is:

where:

  • is the minor obtained by deleting row and column
  • is called the cofactor

You can expand along any row or any column — pick one with zeros to simplify.

Step-by-Step Example

Let’s compute the determinant of:

Row 3 has a zero in position (3,1), so let’s expand along row 3 to minimize calculations.

Expansion Along Row 3

Breaking this down:

  • :
  • :
  • :

Computing the Minors

For (delete row 3, column 2):

For (delete row 3, column 3):

Final Calculation

Alternative: Expansion Along Column 2

Since column 2 has a zero at position (1,2), we could also expand along column 2:

Only non-zero terms:

  • :
  • :

Result:

Key Points

  1. Choice matters: Pick rows/columns with the most zeros
  2. Sign pattern: creates a checkerboard pattern:

Property of Determinants

  1. Transpose:

  2. Inverse: (when is invertible)

  3. Scalar multiplication: for matrix

  4. Identity:

  5. Triangular matrices: Determinant equals product of diagonal entries

  1. Product rule:

    • Matrix multiplication itself is NOT commutative. In general, .
    • The determinant of a matrix is a single number (a scalar). Scalar multiplication IS commutative.

The Gauß Algorithm (Gaussian Elimination)

The Gauß algorithm (Gaussian elimination) is a systematic method for solving systems of linear equations by transforming the coefficient matrix into upper triangular form using elementary row operations.

An upper triangular matrix has all zeros below the main diagonal:

where represents any value (including zero).

  1. Choose pivot: Select the leftmost non-zero entry in the first row
  2. Eliminate below: Use row operations to create zeros below the pivot
  3. Move to next row/column: Repeat for subsequent rows
  4. Continue until upper triangular form is achieved

Example

Transform this matrix to upper triangular form:

Step 1: First column

Eliminate below pivot (2):

:

:

Step 2: Second column

Eliminate below second pivot ():

:

Once in upper triangular form, we can calculate Determinant by:

(product of diagonal entries, with sign depending on number of row swaps)


Inverse of a 2×2 Matrix

Given the matrix

its inverse exists if and only if , where

When , the inverse is

Adjugate and Cofactor Matrix

Minor Matrix

For an square matrix , the minor of entry is the determinant of the submatrix obtained by deleting the -th row and -th column of .

For

The minor (delete row 1, column 2):

The cofactor of is defined as:

The factor creates a “checkerboard” pattern of signs:

The cofactor matrix of is the matrix whose entries are the cofactors:

The adjugate matrix (or adjoint) is the transpose of the cofactor matrix:

Inverse Formula

When is invertible:

Cofactors of a 2×2 Matrix

For , the cofactors are:

EntrySign factorMinorCofactor

For a matrix:

  • Cofactor matrix

  • Adjugate

The inverse of matrix exists if and only if . When , the matrix is said to be singular or non-invertible.


Properties of Matrix Inverses and Transposes

Inverse Properties

  1. Identity Property

  2. Inverse of a Product

  3. Inverse of a Sum (Inequality)

Transpose Properties

  1. Double Transpose

  2. Transpose of a Product

  3. Transpose of a Sum


Row Echelon Form (REF) and Gaussian Elimination

A matrix is in row-echelon form (REF) if:

  1. All nonzero rows are above any rows of all zeros
  2. In each nonzero row, the first nonzero entry (called the leading entry or pivot) is to the right of the leading entry in the row above
  3. All entries below each pivot are zero

Permitted Row Operations

These operations preserve the solution set:

  • Row swap:
  • Scalar multiplication: where
  • Row addition:

2. How to Compute REF (Gaussian Elimination)

Given an augmented matrix :

  1. Choose a pivot column (leftmost column with a nonzero entry below the current working row). If the top candidate is 0, swap with a lower row that has a nonzero entry there
  2. Create zeros below the pivot using row replacements
  3. Move one row down and one column right; repeat until every pivot has zeros beneath it
  4. The resulting matrix is in REF. Solve by back-substitution

Example: Solving a System by REF

System of Equations

Augmented Matrix

Step 1: Choose Pivot in Column 1

Swap :

Step 2: Zero Out Below Pivot in Column 1

Apply operations:

Result:

Step 3: Pivot in Column 3

Use as pivot row for column 3. Apply :

Step 4: Final REF and Consistency Condition

Apply :

This is in REF.


The last row encodes , so the system is consistent if and only if

Back‑Substitution (when a = −1)

With , discard the zero row. Optionally scale the pivot rows for easier reading:

Pivot equations:

Free variables: Columns 2 and 5 have no pivots.

Let , .

Solution set (parametric form):

  • The system effectively has only three independent equations (from the first three non-zero rows) for five variables ().

  • The rank of the coefficient matrix is at most 3 (since there are three non-zero rows), and the augmented matrix has the same rank because the last row contributes no new information.

  • With five variables and three independent equations, there are free variables, which is why the solution has two parameters ( and ) in the general solution


Using REF to Compute a Particular Solution (Column-Space View)

The row-echelon form simplifies the process of determining a particular solution. To find a particular solution:

  1. Express the right-hand side of the equation system using the pivot columns:

    where () are the pivot columns.

  2. The coefficients are determined most easily by:

    • Starting with the rightmost pivot column
    • Working systematically from right to left
  • Setting all free variables to zero produces a particular solution whose nonzero entries occur only in the pivot (basic) positions and equal the coefficients above.

This approach leverages the triangular structure of the row-echelon form to solve for the coefficients through back-substitution, making the calculation process more straightforward and systematic.

In the previous example, we need to find such that:

Solution Process

Working from right to left:

  • From the third row:
  • From the second row:
  • From the first row:

Final Particular Solution

When assembling the complete solution, we must include the non-pivot columns (for which coefficients are implicitly set to 0). Therefore, the particular solution is:

where the zeros in positions 2 and 5 correspond to the non-pivot columns.


Reduced Row-Echelon Form (RREF)

A matrix (or the coefficient matrix of a linear system) is in reduced row‑echelon form (also called row‑reduced echelon form or row canonical form) if:

  1. It is in row‑echelon form (REF).
  2. Every pivot (leading entry) equals .
  3. Each pivot is the only nonzero entry in its column.

♢ Gauss-Jordan elimination is an extension of Gaussian elimination that continues the process to produce reduced row-echelon form (RREF).


Finding Solutions to Ax = 0 Using Pivot and Non-Pivot Columns

The key idea for finding the solutions of is to look at the non-pivot columns, which we will need to express as a (linear) combination of the pivot columns.

Consider the matrix:

which is already in RREF.

1. Identify Pivot and Non-Pivot Columns

Pivots appear at the first nonzero entry of each nonzero row:

  • Row 1 pivot → column 1
  • Row 2 pivot → column 3
  • Row 3 pivot → column 4

Therefore:

  • Pivot columns: 1, 3, 4
  • Non-pivot columns: 2, 5

2. Express Non-Pivot Columns as Linear Combinations of Pivot Columns

Write the column vectors (columns of ) explicitly:

Now express the non-pivot columns in terms of the pivots :

Column 2:

Column 5:

3. Use This to Solve

Write . The homogeneous equation is the vector equation:

Substitute the expressions for and :

Collect coefficients of the pivot columns :

Because are linearly independent (they are standard basis vectors in ), each coefficient must be zero:

4. Parameterize the Solution (Free Variables)

Non-pivot columns correspond to free variables. Let:

Then from the three equations above:

So the general solution to is:


The Null Space (Kernel) of a Matrix

The null space of a matrix , denoted (short for “kernel of ”), is the set of all vectors in the domain of such that:

Mathematical Definition

where is an matrix, and is an vector.

Interpretation

The null space consists of all solutions to the homogeneous system of linear equations . These solutions represent vectors that, when multiplied by , result in the zero vector. Geometrically, the null space describes the set of vectors that are “mapped to zero” by the linear transformation represented by .

Consider the matrix:

This is a matrix, so , and the system represents three equations with five unknowns. The null space is the set of all vectors that satisfy this system.

The general solution to is:

Basis for the Null Space

The null space consists of all linear combinations of the vectors:

These two vectors form a basis for the null space, and the null space is two-dimensional because there are two free variables ( and ).

What Does Represent?

Algebraically

The null space contains all solutions to the homogeneous system. In this case, it describes all vectors that satisfy:

Geometrically

If we think of as a linear transformation from , the null space is the set of all vectors in that are mapped to the zero vector in . In this case, is a two-dimensional subspace of (a “plane” in 5-dimensional space) spanned by the basis vectors above.

Dimension of the Null Space

The dimension of , called the nullity, equals the number of free variables (non-pivot columns). Here:

  • Total columns: 5
  • Pivot columns: 3
  • Non-pivot columns:
  • Therefore, nullity = 2

Importance of the Null Space

1. Solutions to Homogeneous Systems

  • If : only the trivial solution exists
  • If contains non-zero vectors: infinitely many solutions exist

2. Linear Independence

  • If : columns of are linearly independent
  • If is non-trivial: columns of are linearly dependent

3. Rank-Nullity Theorem

For our example:

  • (number of pivot columns)
  • (dimension of null space)
  • (number of columns)

Mechanical Method for Constructing Null Space Basis from RREF : The Free‑Variable Identity‑Block Pattern

This method leverages the structure of the RREF to systematically find the solutions to the homogeneous system .

Short Algorithm (Mechanical Recipe)

Step-by-Step Method

  1. Identify pivot and free columns

    • Pivot columns: those with a leading 1 in each non-zero row of RREF
    • Free columns: those without pivots (corresponding to free variables)
    • Let be in RREF with pivot columns indexed by , free columns indexed by , where .
  2. Construct basis vectors For each free variable:

    • Set that free variable to 1 and all other free variables to 0
    • For each pivot variable, set its value to the negative of the coefficient of the free variable in that pivot row
    • The resulting vector is a basis vector for the null space
  3. Form the basis matrix

    • Collect coefficients of free variables in pivot rows into matrix
    • Construct null space basis matrix
  • Each pivot row has the form which defines the coefficient matrix of the free variables in the pivot equations.

For an matrix in RREF with pivot columns and free columns:

Setup

  • Pivot indices: (pivot variables)
  • Free indices: (free variables)

Construction

  1. Form the matrix :

    • Entry = coefficient of -th free variable (column ) in -th pivot row
  2. Null space basis matrix:

    where:

    • Top rows: pivot variables (negated coefficients from )
    • Bottom rows: free variables (identity matrix )
    • Rows arranged to match original variable order

Each column of is a basis vector for .


Consider the matrix:

This is a matrix in RREF. We seek all such that .

Step 1: Identify Pivot and Free Columns

Pivot columns:

  • Row 1, column 1 ()
  • Row 2, column 3 ()
  • Row 3, column 4 ()

Therefore, pivot columns are 1, 3, and 4, corresponding to variables .

Free columns: Columns 2 and 5 have no pivots, corresponding to free variables and . There are free variables.

Step 2: Read Coefficients of Free Variables

The pivot rows give equations involving free variables and :

  • Row 1: → Coefficients of are
  • Row 2: → Coefficients of are
  • Row 3: → Coefficients of are

Collect these into matrix :

Step 3: Construct Null Space Basis Matrix

The null space basis matrix is:

Reordering rows to match variable order :

Result: Basis Vectors

The columns of are the basis vectors:

These correspond to:

  • For : Set , , then , ,
  • For : Set , , then , ,

The null space is:


The Minus-1 Trick for Homogeneous Systems

The Minus-1 Trick is a practical method for reading out solutions of a homogeneous system of linear equations , where , .

We assume that is in reduced row-echelon form without any rows that contain only zeros

The Augmentation Process

  1. Extend the matrix: We extend this matrix to an matrix by adding rows

  2. Form of added rows: Each added row has the form: where is placed in a non-pivot column position

  3. Result: The diagonal of the augmented matrix contains either 1 or

Key Result

The columns of that contain as pivots are solutions of the homogeneous system .

Example

Given

We now augment this matrix to a 5 × 5 matrix by adding rows of the form [ 0 · · · 0 −1 0 · · · 0 ] at the places where the pivots on the diagonal are missing and obtain

From the augmented matrix , we can immediately read out the solutions of by taking the columns of which contain on the diagonal.

For :


Comparison: Standard Method vs. Minus-1 Trick

Standard "Identity Block" Form

If (after permuting columns so pivots come first) we have:

then the system becomes:

where:

  • are the pivot variables
  • are the free variables

Thus the null space basis is packaged in the matrix:

Minus-1 Trick Form

When you extend to the augmented with ” pivots” for free variables, the free-variable columns in look like:

So the Minus-1 trick produces:

Since multiplying a basis vector by does not change the span, both methods describe exactly the same null space :


Calculating the Inverse Using Gaussian Elimination

To compute the inverse of , we need to find a matrix that satisfies . Then, .

We can write this as a set of simultaneous linear equations , where we solve for .

Using augmented matrix notation for a compact representation:

This means that if we bring the augmented equation system into reduced row-echelon form, we can read out the inverse on the right-hand side of the equation system.

If the left block cannot be reduced to (i.e., a zero row appears), then is singular and does not exist.

Key insight: Determining the inverse of a matrix is equivalent to solving systems of linear equations.

Example

To determine the inverse of:

Step 1: Set up the augmented matrix

Step 2: Apply Gaussian elimination to reach RREF

Step 3: Read the inverse from the right-hand side

The desired inverse is given as:


Matrix Inverse and Solving Linear Systems

For a matrix , the inverse is defined so that:

where is the identity matrix of size .

must be square (), because only then do the dimensions match on both sides for both multiplications.

If is square () and invertible (i.e., ), then:

This is the clean formula solution for solving linear systems.

Non-Invertible Square Matrices

If is not invertible (determinant = 0, or rows/columns are linearly dependent), then does not exist. In that case:

  • The system may have no solution (inconsistent system), or
  • The system may have infinitely many solutions (dependent system)

Non-Square Matrices

If is not square ( with ), Then the usual inverse does not make sense, because .

Generalized Inverses

For non-square or singular matrices, mathematicians define generalized inverses.

(a) Left Inverse

If is tall () and has full column rank (The columns are linearly independent) (rank = ), then:

is invertible. We define the left inverse:

which satisfies:

So it behaves like an inverse on the left.

Why This Formula?

Requirement: A left inverse of must satisfy .

Construction:

  1. is always an symmetric matrix
  2. If has full column rank, then is invertible
  3. Let’s verify: ✓

Intuition:

  • The factor projects things back into the column space of
  • Multiplying by rescales it correctly so the final effect is identity on
  • This construction comes directly from the normal equations in least squares: which are solved by:

Number of Left Inverses for a Matrix

A left inverse of is a matrix such that

The number of left inverses a matrix has depends entirely on its columns. For an matrix where , there are two possible scenarios:

Case 1: Columns are Linearly Independent

If the matrix has full column rank (all columns are linearly independent), then it has infinitely many left inverses.

The Pseudoinverse

One of these is the most important and is called the pseudoinverse:

General Form

You can create all other left inverses by adding any matrix that satisfies :

where and is still a left inverse.

Example

Consider this matrix (its single column is inherently independent):

One left inverse:

Check:

Another left inverse:

Check:

You could replace 5 with any real number, giving infinitely many left inverses.

Case 2: Columns are Linearly Dependent

If the columns are linearly dependent (not full column rank), then no left inverse exists.

Sylvester’s rank inequality: For compatible matrices,

(b) Right Inverse

If is wide () and has full row rank (rank = ), then:

is invertible. We define the right inverse:

  • Check: Thus “undoes” multiplication by from the right.

(c) Moore-Penrose Pseudoinverse

Least Squares Method (overdetermined case)

Often in applications (statistics, machine learning, data fitting), is tall (), meaning we have more equations than unknowns. In that case, the system is usually inconsistent — there is no exact such that .

Since we cannot solve exactly, we instead look for an that makes as close as possible to .

We cannot satisfy all equations, but we can find a vector that makes the residual

as small as possible in the Euclidean norm.

So we solve:

Formally:

This is the least squares problem: we want the that minimizes the squared error between and .

Derivation (normal equations) :

The error is:

Then

Setting gives the normal equations

Solution

  • If has full column rank (), then is invertible and the unique minimizer is

    Equivalently, (pseudoinverse).

  • This is called the least-squares solution because it minimizes the squared error between Ax and 𝑏

  • is the projection of onto the column space of


Minimum-norm solution (underdetermined case)

Underdetermined (wide) system: too many possible solutions, so we pick the one that is shortest in length. That’s the minimum-norm solution.

Among all possible solutions, we select the one with the smallest Euclidean norm:

If (A) has full row rank, the solution can be expressed as:

where denotes the Moore-Penrose pseudoinverse of (A).

This is called the minimum-norm solution because it picks the shortest vector among the infinitely many possible solutions.


Rank-deficient case

For a matrix the rank of (A) is defined as the maximum number of linearly independent columns (or rows) of (A).

If we say that (A) is rank-deficient.

Intuitively, this means that some columns (or rows) are linearly dependent on others. Consequently:

  • Some directions in the solution space are “free,” leading to infinitely many solutions in the underdetermined case.
  • Even in overdetermined least-squares problems, the minimizer may not be unique, because one can move along null-space directions without changing (Ax).

The pseudoinverse defined via the singular value decomposition (SVD), always exists and provides:

  • Overdetermined case: Multiple least-squares minimizers exist. The pseudoinverse selects the one with minimum norm.
  • Underdetermined case: Infinitely many solutions exist. The pseudoinverse selects the minimum-norm solution.

In this case, the Moore–Penrose pseudoinverse picks out one special solution:

namely the one with minimum Euclidean norm among all least-squares solutions. This is called the minimum-norm least-squares solution, because it not only minimizes but also selects the solution with the smallest Euclidean norm among all possible least-squares solutions.


The Moore-Penrose pseudoinverse is a generalization that:

  • Always exists, for any
  • No matter what shape or rank A has is defined.
  • Reduces to the usual inverse if is square and invertible

Formula (when has full column rank):

  • Full column rank ():

Formula (when has full row rank):

  • Full row rank ():

The pseudoinverse is particularly important in data science, statistics, and machine learning for least squares regression.

For reasons of numerical precision it is generally not recommended to compute the inverse or pseudo-inverse. Forming requires a matrix–matrix multiplication, which is computationally expensive for large . Furthermore, computing the inverse adds additional cost.

How is it Computed?

The most robust way is via the Singular Value Decomposition (SVD):

where are orthogonal and is diagonal with nonnegative entries (the singular values).

Then:

where is obtained by inverting each nonzero singular value and transposing the shape of .

This construction guarantees:

  • Always exists
  • Stable and numerically reliable
  • Encodes rank and null space information cleanly

The pseudoinverse is a matrix you can always use in place of an inverse.


Geometric view:

Define the projectors

Then projects onto and projects onto , both orthogonally:

For , the residual satisfies .


Gaussian elimination is not only for solving . The same algorithm underpins many core operations in linear algebra:

  • Determinants: You can compute by reducing to triangular form and multiplying the pivots.
  • Independence: By row reducing a set of vectors, you can check if they are linearly independent.
  • Inverses: By augmenting with and performing row reduction, one obtains .
  • Rank: The number of pivots (nonzero rows in echelon form) gives .
  • Bases: From the reduced row echelon form (RREF), one can extract bases for the column space, row space, or null space.

Gaussian elimination has a computational cost of about:

operations for an system.

  • In the first column, you eliminate entries below the pivot. That means you update about rows, and each update touches about entries of that row.
    → Roughly operations.

  • In the second column, the active submatrix is now of size . Work there is about .

  • Next, , and so on, until the last few pivots are trivial.

The total cost is:

And recall (or prove) the identity:

Thus, the total operation count is on the order of:

  • For millions of variables: becomes astronomically expensive in both time and memory

Iterative Methods

Instead, we turn to iterative methods, which don’t try to solve the whole system at once, but refine an approximation step by step.

General Idea

where and are chosen so that the approximation converges to the true solution .

  • At each step, the residual: gets smaller

  • The error norm shrinks with iterations

Examples of Iterative Methods

  • Jacobi method
  • Gauss-Seidel method
  • Successive over-relaxation (SOR)
  • Conjugate gradient (CG) and other Krylov subspace methods

These methods scale much better for large, sparse systems because:

  • They don’t require full elimination

  • They exploit sparsity (lots of zeros in )

  • Often, convergence is reached in far fewer than steps

    **Gaussian elimination** = exact, finite, but costly → good for moderate-size problems
    
    **Iterative methods** = approximate but scalable → essential for large systems in science/engineering (e.g., fluid dynamics, machine learning)
    

Subspace

Subspaces are incredibly important in linear algebra because they reveal the underlying structure of problems.

The requirement that subspaces pass through the origin isn't arbitrary—it's what makes linear algebra "linear." The origin serves as the universal reference point, the additive identity, ensuring that all linear operations remain consistent and well-defined within the subspace

They Are Self-Contained Vector Spaces

A subspace isn’t just a random collection of vectors; it’s a complete vector space nested inside a larger one. This means:

  • You can add, subtract, and scale vectors within the subspace
  • The result will always stay within that same subspace
  • This self-contained, predictable behavior makes them mathematically tractable

They Describe Solutions to Systems of Equations

The set of all solutions to any homogeneous system of linear equations (like ) always forms a subspace called the null space.

Any matrix A multiplied by the zero vector will always result in the zero vector. This means that X=0 (the origin) is always a solution to the system. Since the origin is always one of the solutions, any line or plane that represents the entire set of solutions must contain this point.

This gives a powerful geometric interpretation:

  • Solutions aren’t just a list of numbers
  • They form a line, plane, or higher-dimensional space through the origin
  • The dimension tells you the degrees of freedom in the solution

They Define the Fundamental Properties of a Matrix

The most important properties of a matrix are described by its four fundamental subspaces:

SubspaceDefinitionWhat it tells us
Column SpaceAll possible outputs The “range” or “image” of the matrix. Any output of Ax must lie on this plane.
Null SpaceAll where What information is “lost” in the transformation
Row SpaceAll linear combinations of rowsThe constraints the matrix imposes
Left Null SpaceAll where Orthogonal complement to column space

The Four Fundamental Subspaces: A Concrete Example

Consider the following matrix :

Notice that the third column is the sum of the first two columns. This dependency is what will create interesting subspaces.

To find the basis for all four subspaces, the most effective method is to reduce the matrix to its Row-Reduced Echelon Form (RREF).

The RREF of is:

Row operations don’t change the row space but row operations change the columns, so the columns of the RREF do not generally span the same column space

Now we can find a basis for each subspace.


1. Column Space, - (The Outputs)

How to find

Look at the pivot columns in the RREF (). The pivots are in columns 1 and 2. The basis for the column space is the corresponding columns from the original matrix .

Basis

Geometric Meaning

The column space is a plane in 3D space defined by these two vectors.

What it tells us

Any output of must lie on this plane. The equation has a solution only if is a vector on this plane.

For example, the vector is not on this plane, so there is no that can produce it. You can’t reach it.


2. Null Space (Right null space), - (The Inputs that become Zero)

How to find

Solve the equation .

This gives us the equations:

  • is a free variable. Let’s call it .

The solution vector is .

Basis

Geometric Meaning

The null space is a line in 3D space passing through the origin.

What it tells us

Any input vector on this line is “lost” or “squashed” to the zero vector by the transformation . This is the information that the matrix loses.

For example, the vector and the vector both get sent to .


3. Row Space, - (The “Active” Inputs)

How to find

The non-zero rows of the RREF () form a basis for the row space.

Basis

Geometric Meaning

The row space is a plane in 3D space.

What it tells us

This plane represents the domain of “effective” inputs. Any part of an input vector that is orthogonal (perpendicular) to this plane will be in the null space and get sent to zero.

Key Connection

Notice that the basis vector for the Null Space is orthogonal to both basis vectors for the Row Space:

This is always true: The Null Space and the Row Space are orthogonal complements.


4. Left Null Space, - (The Constraints on the Outputs)

How to find

This is the null space of . We can find it by finding the dependencies in the rows of during elimination. In our RREF calculation, the last row became all zeros by the operation R3 → R3 + R1 - 2R2 (after a couple of steps). This relationship gives us the basis vector.

A more direct way is to solve .

Basis

Geometric Meaning

This is a line in 3D space.

What it tells us

This space defines the constraints for a vector to be in the column space. A vector is a valid output if and only if it is orthogonal to the left null space.

Key Connection

The basis vector for the Left Null Space is orthogonal to both basis vectors for the Column Space:

This tells us that for to have a solution, must satisfy . This is the equation of the plane we found for the column space!


Fundamental Theorem of Linear Algebra

  • (2 + 1 = 3)
  • (2 + 1 = 3)

Understanding these subspaces allows us to completely understand the behavior of the linear transformation that the matrix represents.


A subspace must be a “flat” object that passes through the origin.

It can be:

  • A point (just the origin)
  • A line through the origin
  • A plane through the origin
  • All of

But it cannot be curved, shifted away from the origin, or have boundaries.

To determine if a set is a subspace of , you must check if it meets three conditions:

The Three Conditions

1. Contains the Zero Vector

The set must contain

2. Closed Under Addition

If you add any two vectors from the set, the result is still in the set.

  • If , then

3. Closed Under Scalar Multiplication

If you multiply any vector from the set by a scalar, the result is still in the set.

  • If and , then

Examples of Subspaces

Example 1: A Plane Through the Origin

Check 1 (Zero vector):

  • Does ? ✓ Yes

Check 2 (Addition):

  • If , then and
  • For : ✓

Check 3 (Scalar multiplication):

  • If and : ✓

Conclusion: This is a subspace (a plane through the origin).

Example 2: A Line Through the Origin

All three conditions are satisfied → This is a subspace.

Examples of Non-Subspaces

Example 1: Plane Not Through Origin

Check 1: Does satisfy ?

  • ✗

Conclusion: Not a subspace (fails condition 1).

Example 2: Upper Hemisphere

Check 2: Take and

Both are in , but

Check: ✗

Conclusion: Not a subspace (fails closure under addition).