Determinants
The determinant is a scalar value that encodes both the volume and orientation of the geometric object (parallelepiped) formed by the rows or columns of a matrix.
Determinant in Two Dimensions (Area)
Suppose we have two vectors in :
If we put these as rows (or columns) of a matrix:
then the determinant is:
Geometric Interpretation
- The parallelogram spanned by and has area
- The sign of indicates orientation:
- Positive: if rotated counterclockwise to
- Negative: if clockwise
So in 2D, the determinant is the signed area of the parallelogram formed by the two vectors.
Determinant in Three Dimensions (Volume)
Consider three vectors in :
If we arrange them as rows of a matrix , then:
- gives the volume of the parallelepiped formed by
General Case in
For an matrix:
- gives the -dimensional volume (hypervolume) of the parallelepiped spanned by the rows (or columns)
- The sign encodes orientation relative to the standard basis
For any square matrix :
Matrix Notation for Dot Product
The matrix notation is used because the mechanical process of multiplying the matrices and results in the exact same calculation as the dot product .
Suppose we have the column vectors:
1. The Dot Product ()
The standard dot product is calculated by multiplying corresponding components and adding them up:
2. The Matrix Notation ()
First, we find the transpose of , which turns the column vector into a row vector:
Now, we perform standard matrix multiplication of (a matrix) and (a matrix):
The rule for matrix multiplication is “row times column”:
The result is a matrix, which is just a single number (a scalar).
Triangular Matrices and Their Determinants
Lower Triangular Matrix
A lower triangular matrix is a square matrix where all the entries above the main diagonal are zero.
Example of a 4×4 lower triangular matrix:
Upper Triangular Matrix
An upper triangular matrix is the opposite, with all entries below the main diagonal being zero.
Example of a 4×4 upper triangular matrix:
Determinant of a Triangular Matrix
For both upper and lower triangular matrices, the determinant is simply the product of the entries on the main diagonal.
Block Upper Triangular Matrices
Recognizing a block triangular structure is a major computational shortcut, especially for finding determinants and eigenvalues.
Simplifying Determinants
The determinant of a block triangular matrix is simply the product of the determinants of its diagonal blocks.
For example:
Instead of computing a difficult 4×4 determinant, you only need to calculate two simple 2×2 determinants:
The determinant of the full matrix is just their product:
Finding Eigenvalues
The set of eigenvalues of a block triangular matrix is the union of the eigenvalues of its diagonal blocks.
This means you can:
- Find eigenvalues of → get
- Find eigenvalues of → get
- Eigenvalues of =
A matrix can be partitioned into any number of blocks of any size, as long as the partitioning is consistent.
Applications
- Markov Chains: States often group into communicating classes → block structure
- Network Analysis: Communities in networks lead to block patterns
- Differential Equations: Decoupled or weakly coupled systems
- Quantum Mechanics: Block diagonal Hamiltonians for non-interacting subsystems
Row Operations and Determinants
Row operations preserve the linear dependency relationships between the columns.
1. Row Swap
Swapping two rows multiplies the determinant by .
2. Row Addition
Adding a multiple of one row to another does not change the determinant.
3. Row Scaling
Multiplying a row by a non-zero scalar multiplies the determinant by .
Example
Let’s compute the determinant of:
Row Operations to Create Upper Triangular Form
Operation 1: (no change to determinant)
Operation 2: (no change to determinant)
Operation 3: (no change to determinant)
For an upper triangular matrix, the determinant is the product of diagonal elements:
Elementary Matrices
An elementary matrix is a square matrix obtained from the identity matrix by performing exactly one elementary row operation. Pre-multiplying any matrix by an elementary matrix applies that same row operation to the matrix as a whole.
- If is elementary and is any matrix, then is with one row operation applied.
- Right-multiplication by an elementary matrix applies the corresponding column operation: .
Laplace (Cofactor) Expansion for Determinants
For an matrix , the Laplace expansion along row is:
where:
- is the minor obtained by deleting row and column
- is called the cofactor
You can expand along any row or any column — pick one with zeros to simplify.
Step-by-Step Example
Let’s compute the determinant of:
Row 3 has a zero in position (3,1), so let’s expand along row 3 to minimize calculations.
Expansion Along Row 3
Breaking this down:
- :
- :
- :
Computing the Minors
For (delete row 3, column 2):
For (delete row 3, column 3):
Final Calculation
Alternative: Expansion Along Column 2
Since column 2 has a zero at position (1,2), we could also expand along column 2:
Only non-zero terms:
- :
- :
Result:
Key Points
- Choice matters: Pick rows/columns with the most zeros
- Sign pattern: creates a checkerboard pattern:
Property of Determinants
-
Transpose:
-
Inverse: (when is invertible)
-
Scalar multiplication: for matrix
-
Identity:
-
Triangular matrices: Determinant equals product of diagonal entries
-
Product rule:
- Matrix multiplication itself is NOT commutative. In general, .
- The determinant of a matrix is a single number (a scalar). Scalar multiplication IS commutative.
The Gauß Algorithm (Gaussian Elimination)
The Gauß algorithm (Gaussian elimination) is a systematic method for solving systems of linear equations by transforming the coefficient matrix into upper triangular form using elementary row operations.
An upper triangular matrix has all zeros below the main diagonal:
where represents any value (including zero).
- Choose pivot: Select the leftmost non-zero entry in the first row
- Eliminate below: Use row operations to create zeros below the pivot
- Move to next row/column: Repeat for subsequent rows
- Continue until upper triangular form is achieved
Example
Transform this matrix to upper triangular form:
Step 1: First column
Eliminate below pivot (2):
:
:
Step 2: Second column
Eliminate below second pivot ():
:
Once in upper triangular form, we can calculate Determinant by:
(product of diagonal entries, with sign depending on number of row swaps)
Inverse of a 2×2 Matrix
Given the matrix
its inverse exists if and only if , where
When , the inverse is
Adjugate and Cofactor Matrix
Minor Matrix
For an square matrix , the minor of entry is the determinant of the submatrix obtained by deleting the -th row and -th column of .
For
The minor (delete row 1, column 2):
The cofactor of is defined as:
The factor creates a “checkerboard” pattern of signs:
The cofactor matrix of is the matrix whose entries are the cofactors:
The adjugate matrix (or adjoint) is the transpose of the cofactor matrix:
Inverse Formula
When is invertible:
Cofactors of a 2×2 Matrix
For , the cofactors are:
| Entry | Sign factor | Minor | Cofactor |
|---|---|---|---|
For a matrix:
-
Cofactor matrix
-
Adjugate
The inverse of matrix exists if and only if . When , the matrix is said to be singular or non-invertible.
Properties of Matrix Inverses and Transposes
Inverse Properties
-
Identity Property
-
Inverse of a Product
-
Inverse of a Sum (Inequality)
Transpose Properties
-
Double Transpose
-
Transpose of a Product
-
Transpose of a Sum
Row Echelon Form (REF) and Gaussian Elimination
A matrix is in row-echelon form (REF) if:
- All nonzero rows are above any rows of all zeros
- In each nonzero row, the first nonzero entry (called the leading entry or pivot) is to the right of the leading entry in the row above
- All entries below each pivot are zero
Permitted Row Operations
These operations preserve the solution set:
- Row swap:
- Scalar multiplication: where
- Row addition:
2. How to Compute REF (Gaussian Elimination)
Given an augmented matrix :
- Choose a pivot column (leftmost column with a nonzero entry below the current working row). If the top candidate is 0, swap with a lower row that has a nonzero entry there
- Create zeros below the pivot using row replacements
- Move one row down and one column right; repeat until every pivot has zeros beneath it
- The resulting matrix is in REF. Solve by back-substitution
Example: Solving a System by REF
System of Equations
Augmented Matrix
Step 1: Choose Pivot in Column 1
Swap :
Step 2: Zero Out Below Pivot in Column 1
Apply operations:
Result:
Step 3: Pivot in Column 3
Use as pivot row for column 3. Apply :
Step 4: Final REF and Consistency Condition
Apply :
This is in REF.
The last row encodes , so the system is consistent if and only if
Back‑Substitution (when a = −1)
With , discard the zero row. Optionally scale the pivot rows for easier reading:
Pivot equations:
Free variables: Columns 2 and 5 have no pivots.
Let , .
Solution set (parametric form):
-
The system effectively has only three independent equations (from the first three non-zero rows) for five variables ().
-
The rank of the coefficient matrix is at most 3 (since there are three non-zero rows), and the augmented matrix has the same rank because the last row contributes no new information.
-
With five variables and three independent equations, there are free variables, which is why the solution has two parameters ( and ) in the general solution
Using REF to Compute a Particular Solution (Column-Space View)
The row-echelon form simplifies the process of determining a particular solution. To find a particular solution:
-
Express the right-hand side of the equation system using the pivot columns:
where () are the pivot columns.
-
The coefficients are determined most easily by:
- Starting with the rightmost pivot column
- Working systematically from right to left
- Setting all free variables to zero produces a particular solution whose nonzero entries occur only in the pivot (basic) positions and equal the coefficients above.
This approach leverages the triangular structure of the row-echelon form to solve for the coefficients through back-substitution, making the calculation process more straightforward and systematic.
In the previous example, we need to find such that:
Solution Process
Working from right to left:
- From the third row:
- From the second row:
- From the first row:
Final Particular Solution
When assembling the complete solution, we must include the non-pivot columns (for which coefficients are implicitly set to 0). Therefore, the particular solution is:
where the zeros in positions 2 and 5 correspond to the non-pivot columns.
Reduced Row-Echelon Form (RREF)
A matrix (or the coefficient matrix of a linear system) is in reduced row‑echelon form (also called row‑reduced echelon form or row canonical form) if:
- It is in row‑echelon form (REF).
- Every pivot (leading entry) equals .
- Each pivot is the only nonzero entry in its column.
♢ Gauss-Jordan elimination is an extension of Gaussian elimination that continues the process to produce reduced row-echelon form (RREF).
Finding Solutions to Ax = 0 Using Pivot and Non-Pivot Columns
The key idea for finding the solutions of is to look at the non-pivot columns, which we will need to express as a (linear) combination of the pivot columns.
Consider the matrix:
which is already in RREF.
1. Identify Pivot and Non-Pivot Columns
Pivots appear at the first nonzero entry of each nonzero row:
- Row 1 pivot → column 1
- Row 2 pivot → column 3
- Row 3 pivot → column 4
Therefore:
- Pivot columns: 1, 3, 4
- Non-pivot columns: 2, 5
2. Express Non-Pivot Columns as Linear Combinations of Pivot Columns
Write the column vectors (columns of ) explicitly:
Now express the non-pivot columns in terms of the pivots :
Column 2:
Column 5:
3. Use This to Solve
Write . The homogeneous equation is the vector equation:
Substitute the expressions for and :
Collect coefficients of the pivot columns :
Because are linearly independent (they are standard basis vectors in ), each coefficient must be zero:
4. Parameterize the Solution (Free Variables)
Non-pivot columns correspond to free variables. Let:
Then from the three equations above:
So the general solution to is:
The Null Space (Kernel) of a Matrix
The null space of a matrix , denoted (short for “kernel of ”), is the set of all vectors in the domain of such that:
Mathematical Definition
where is an matrix, and is an vector.
Interpretation
The null space consists of all solutions to the homogeneous system of linear equations . These solutions represent vectors that, when multiplied by , result in the zero vector. Geometrically, the null space describes the set of vectors that are “mapped to zero” by the linear transformation represented by .
Consider the matrix:
This is a matrix, so , and the system represents three equations with five unknowns. The null space is the set of all vectors that satisfy this system.
The general solution to is:
Basis for the Null Space
The null space consists of all linear combinations of the vectors:
These two vectors form a basis for the null space, and the null space is two-dimensional because there are two free variables ( and ).
What Does Represent?
Algebraically
The null space contains all solutions to the homogeneous system. In this case, it describes all vectors that satisfy:
Geometrically
If we think of as a linear transformation from , the null space is the set of all vectors in that are mapped to the zero vector in . In this case, is a two-dimensional subspace of (a “plane” in 5-dimensional space) spanned by the basis vectors above.
Dimension of the Null Space
The dimension of , called the nullity, equals the number of free variables (non-pivot columns). Here:
- Total columns: 5
- Pivot columns: 3
- Non-pivot columns:
- Therefore, nullity = 2
Importance of the Null Space
1. Solutions to Homogeneous Systems
- If : only the trivial solution exists
- If contains non-zero vectors: infinitely many solutions exist
2. Linear Independence
- If : columns of are linearly independent
- If is non-trivial: columns of are linearly dependent
3. Rank-Nullity Theorem
For our example:
- (number of pivot columns)
- (dimension of null space)
- (number of columns)
Mechanical Method for Constructing Null Space Basis from RREF : The Free‑Variable Identity‑Block Pattern
This method leverages the structure of the RREF to systematically find the solutions to the homogeneous system .
Short Algorithm (Mechanical Recipe)
Step-by-Step Method
-
Identify pivot and free columns
- Pivot columns: those with a leading 1 in each non-zero row of RREF
- Free columns: those without pivots (corresponding to free variables)
- Let be in RREF with pivot columns indexed by , free columns indexed by , where .
-
Construct basis vectors For each free variable:
- Set that free variable to 1 and all other free variables to 0
- For each pivot variable, set its value to the negative of the coefficient of the free variable in that pivot row
- The resulting vector is a basis vector for the null space
-
Form the basis matrix
- Collect coefficients of free variables in pivot rows into matrix
- Construct null space basis matrix
- Each pivot row has the form which defines the coefficient matrix of the free variables in the pivot equations.
For an matrix in RREF with pivot columns and free columns:
Setup
- Pivot indices: (pivot variables)
- Free indices: (free variables)
Construction
-
Form the matrix :
- Entry = coefficient of -th free variable (column ) in -th pivot row
-
Null space basis matrix:
where:
- Top rows: pivot variables (negated coefficients from )
- Bottom rows: free variables (identity matrix )
- Rows arranged to match original variable order
Each column of is a basis vector for .
Consider the matrix:
This is a matrix in RREF. We seek all such that .
Step 1: Identify Pivot and Free Columns
Pivot columns:
- Row 1, column 1 ()
- Row 2, column 3 ()
- Row 3, column 4 ()
Therefore, pivot columns are 1, 3, and 4, corresponding to variables .
Free columns: Columns 2 and 5 have no pivots, corresponding to free variables and . There are free variables.
Step 2: Read Coefficients of Free Variables
The pivot rows give equations involving free variables and :
- Row 1: → Coefficients of are
- Row 2: → Coefficients of are
- Row 3: → Coefficients of are
Collect these into matrix :
Step 3: Construct Null Space Basis Matrix
The null space basis matrix is:
Reordering rows to match variable order :
Result: Basis Vectors
The columns of are the basis vectors:
These correspond to:
- For : Set , , then , ,
- For : Set , , then , ,
The null space is:
The Minus-1 Trick for Homogeneous Systems
The Minus-1 Trick is a practical method for reading out solutions of a homogeneous system of linear equations , where , .
We assume that is in reduced row-echelon form without any rows that contain only zeros
The Augmentation Process
-
Extend the matrix: We extend this matrix to an matrix by adding rows
-
Form of added rows: Each added row has the form: where is placed in a non-pivot column position
-
Result: The diagonal of the augmented matrix contains either 1 or
Key Result
The columns of that contain as pivots are solutions of the homogeneous system .
Example
Given
We now augment this matrix to a 5 × 5 matrix by adding rows of the form [ 0 · · · 0 −1 0 · · · 0 ] at the places where the pivots on the diagonal are missing and obtain
From the augmented matrix , we can immediately read out the solutions of by taking the columns of which contain on the diagonal.
For :
Comparison: Standard Method vs. Minus-1 Trick
Standard "Identity Block" Form
If (after permuting columns so pivots come first) we have:
then the system becomes:
where:
- are the pivot variables
- are the free variables
Thus the null space basis is packaged in the matrix:
Minus-1 Trick Form
When you extend to the augmented with ” pivots” for free variables, the free-variable columns in look like:
So the Minus-1 trick produces:
Since multiplying a basis vector by does not change the span, both methods describe exactly the same null space :
Calculating the Inverse Using Gaussian Elimination
To compute the inverse of , we need to find a matrix that satisfies . Then, .
We can write this as a set of simultaneous linear equations , where we solve for .
Using augmented matrix notation for a compact representation:
This means that if we bring the augmented equation system into reduced row-echelon form, we can read out the inverse on the right-hand side of the equation system.
If the left block cannot be reduced to (i.e., a zero row appears), then is singular and does not exist.
Key insight: Determining the inverse of a matrix is equivalent to solving systems of linear equations.
Example
To determine the inverse of:
Step 1: Set up the augmented matrix
Step 2: Apply Gaussian elimination to reach RREF
Step 3: Read the inverse from the right-hand side
The desired inverse is given as:
Matrix Inverse and Solving Linear Systems
For a matrix , the inverse is defined so that:
where is the identity matrix of size .
must be square (), because only then do the dimensions match on both sides for both multiplications.
If is square () and invertible (i.e., ), then:
This is the clean formula solution for solving linear systems.
Non-Invertible Square Matrices
If is not invertible (determinant = 0, or rows/columns are linearly dependent), then does not exist. In that case:
- The system may have no solution (inconsistent system), or
- The system may have infinitely many solutions (dependent system)
Non-Square Matrices
If is not square ( with ), Then the usual inverse does not make sense, because .
Generalized Inverses
For non-square or singular matrices, mathematicians define generalized inverses.
(a) Left Inverse
If is tall () and has full column rank (The columns are linearly independent) (rank = ), then:
is invertible. We define the left inverse:
which satisfies:
So it behaves like an inverse on the left.
Why This Formula?
Requirement: A left inverse of must satisfy .
Construction:
- is always an symmetric matrix
- If has full column rank, then is invertible
- Let’s verify: ✓
Intuition:
- The factor projects things back into the column space of
- Multiplying by rescales it correctly so the final effect is identity on
- This construction comes directly from the normal equations in least squares: which are solved by:
Number of Left Inverses for a Matrix
A left inverse of is a matrix such that
The number of left inverses a matrix has depends entirely on its columns. For an matrix where , there are two possible scenarios:
Case 1: Columns are Linearly Independent
If the matrix has full column rank (all columns are linearly independent), then it has infinitely many left inverses.
The Pseudoinverse
One of these is the most important and is called the pseudoinverse:
General Form
You can create all other left inverses by adding any matrix that satisfies :
where and is still a left inverse.
Example
Consider this matrix (its single column is inherently independent):
One left inverse:
Check:
Another left inverse:
Check:
You could replace 5 with any real number, giving infinitely many left inverses.
Case 2: Columns are Linearly Dependent
If the columns are linearly dependent (not full column rank), then no left inverse exists.
Sylvester’s rank inequality: For compatible matrices,
(b) Right Inverse
If is wide () and has full row rank (rank = ), then:
is invertible. We define the right inverse:
- Check: Thus “undoes” multiplication by from the right.
(c) Moore-Penrose Pseudoinverse
Least Squares Method (overdetermined case)
Often in applications (statistics, machine learning, data fitting), is tall (), meaning we have more equations than unknowns. In that case, the system is usually inconsistent — there is no exact such that .
Since we cannot solve exactly, we instead look for an that makes as close as possible to .
We cannot satisfy all equations, but we can find a vector that makes the residual
as small as possible in the Euclidean norm.
So we solve:
Formally:
This is the least squares problem: we want the that minimizes the squared error between and .
Derivation (normal equations) :
The error is:
Then
Setting gives the normal equations
Solution
-
If has full column rank (), then is invertible and the unique minimizer is
Equivalently, (pseudoinverse).
-
This is called the least-squares solution because it minimizes the squared error between Ax and 𝑏
-
is the projection of onto the column space of
Minimum-norm solution (underdetermined case)
Underdetermined (wide) system: too many possible solutions, so we pick the one that is shortest in length. That’s the minimum-norm solution.
Among all possible solutions, we select the one with the smallest Euclidean norm:
If (A) has full row rank, the solution can be expressed as:
where denotes the Moore-Penrose pseudoinverse of (A).
This is called the minimum-norm solution because it picks the shortest vector among the infinitely many possible solutions.
Rank-deficient case
For a matrix the rank of (A) is defined as the maximum number of linearly independent columns (or rows) of (A).
If we say that (A) is rank-deficient.
Intuitively, this means that some columns (or rows) are linearly dependent on others. Consequently:
- Some directions in the solution space are “free,” leading to infinitely many solutions in the underdetermined case.
- Even in overdetermined least-squares problems, the minimizer may not be unique, because one can move along null-space directions without changing (Ax).
The pseudoinverse defined via the singular value decomposition (SVD), always exists and provides:
- Overdetermined case: Multiple least-squares minimizers exist. The pseudoinverse selects the one with minimum norm.
- Underdetermined case: Infinitely many solutions exist. The pseudoinverse selects the minimum-norm solution.
In this case, the Moore–Penrose pseudoinverse picks out one special solution:
namely the one with minimum Euclidean norm among all least-squares solutions. This is called the minimum-norm least-squares solution, because it not only minimizes but also selects the solution with the smallest Euclidean norm among all possible least-squares solutions.
The Moore-Penrose pseudoinverse is a generalization that:
- Always exists, for any
- No matter what shape or rank A has is defined.
- Reduces to the usual inverse if is square and invertible
Formula (when has full column rank):
- Full column rank ():
Formula (when has full row rank):
- Full row rank ():
The pseudoinverse is particularly important in data science, statistics, and machine learning for least squares regression.
For reasons of numerical precision it is generally not recommended to compute the inverse or pseudo-inverse. Forming requires a matrix–matrix multiplication, which is computationally expensive for large . Furthermore, computing the inverse adds additional cost.
How is it Computed?
The most robust way is via the Singular Value Decomposition (SVD):
where are orthogonal and is diagonal with nonnegative entries (the singular values).
Then:
where is obtained by inverting each nonzero singular value and transposing the shape of .
This construction guarantees:
- Always exists
- Stable and numerically reliable
- Encodes rank and null space information cleanly
The pseudoinverse is a matrix you can always use in place of an inverse.
Geometric view:
Define the projectors
Then projects onto and projects onto , both orthogonally:
For , the residual satisfies .
Gaussian elimination is not only for solving . The same algorithm underpins many core operations in linear algebra:
- Determinants: You can compute by reducing to triangular form and multiplying the pivots.
- Independence: By row reducing a set of vectors, you can check if they are linearly independent.
- Inverses: By augmenting with and performing row reduction, one obtains .
- Rank: The number of pivots (nonzero rows in echelon form) gives .
- Bases: From the reduced row echelon form (RREF), one can extract bases for the column space, row space, or null space.
Gaussian elimination has a computational cost of about:
operations for an system.
-
In the first column, you eliminate entries below the pivot. That means you update about rows, and each update touches about entries of that row.
→ Roughly operations. -
In the second column, the active submatrix is now of size . Work there is about .
-
Next, , and so on, until the last few pivots are trivial.
The total cost is:
And recall (or prove) the identity:
Thus, the total operation count is on the order of:
- For millions of variables: becomes astronomically expensive in both time and memory
Iterative Methods
Instead, we turn to iterative methods, which don’t try to solve the whole system at once, but refine an approximation step by step.
General Idea
where and are chosen so that the approximation converges to the true solution .
-
At each step, the residual: gets smaller
-
The error norm shrinks with iterations
Examples of Iterative Methods
- Jacobi method
- Gauss-Seidel method
- Successive over-relaxation (SOR)
- Conjugate gradient (CG) and other Krylov subspace methods
These methods scale much better for large, sparse systems because:
-
They don’t require full elimination
-
They exploit sparsity (lots of zeros in )
-
Often, convergence is reached in far fewer than steps
**Gaussian elimination** = exact, finite, but costly → good for moderate-size problems **Iterative methods** = approximate but scalable → essential for large systems in science/engineering (e.g., fluid dynamics, machine learning)
Subspace
Subspaces are incredibly important in linear algebra because they reveal the underlying structure of problems.
The requirement that subspaces pass through the origin isn't arbitrary—it's what makes linear algebra "linear." The origin serves as the universal reference point, the additive identity, ensuring that all linear operations remain consistent and well-defined within the subspace
They Are Self-Contained Vector Spaces
A subspace isn’t just a random collection of vectors; it’s a complete vector space nested inside a larger one. This means:
- You can add, subtract, and scale vectors within the subspace
- The result will always stay within that same subspace
- This self-contained, predictable behavior makes them mathematically tractable
They Describe Solutions to Systems of Equations
The set of all solutions to any homogeneous system of linear equations (like ) always forms a subspace called the null space.
Any matrix A multiplied by the zero vector will always result in the zero vector. This means that X=0 (the origin) is always a solution to the system. Since the origin is always one of the solutions, any line or plane that represents the entire set of solutions must contain this point.
This gives a powerful geometric interpretation:
- Solutions aren’t just a list of numbers
- They form a line, plane, or higher-dimensional space through the origin
- The dimension tells you the degrees of freedom in the solution
They Define the Fundamental Properties of a Matrix
The most important properties of a matrix are described by its four fundamental subspaces:
| Subspace | Definition | What it tells us |
|---|---|---|
| Column Space | All possible outputs | The “range” or “image” of the matrix. Any output of Ax must lie on this plane. |
| Null Space | All where | What information is “lost” in the transformation |
| Row Space | All linear combinations of rows | The constraints the matrix imposes |
| Left Null Space | All where | Orthogonal complement to column space |
The Four Fundamental Subspaces: A Concrete Example
Consider the following matrix :
Notice that the third column is the sum of the first two columns. This dependency is what will create interesting subspaces.
To find the basis for all four subspaces, the most effective method is to reduce the matrix to its Row-Reduced Echelon Form (RREF).
The RREF of is:
Row operations don’t change the row space but row operations change the columns, so the columns of the RREF do not generally span the same column space
Now we can find a basis for each subspace.
1. Column Space, - (The Outputs)
How to find
Look at the pivot columns in the RREF (). The pivots are in columns 1 and 2. The basis for the column space is the corresponding columns from the original matrix .
Basis
Geometric Meaning
The column space is a plane in 3D space defined by these two vectors.
What it tells us
Any output of must lie on this plane. The equation has a solution only if is a vector on this plane.
For example, the vector is not on this plane, so there is no that can produce it. You can’t reach it.
2. Null Space (Right null space), - (The Inputs that become Zero)
How to find
Solve the equation .
This gives us the equations:
- is a free variable. Let’s call it .
The solution vector is .
Basis
Geometric Meaning
The null space is a line in 3D space passing through the origin.
What it tells us
Any input vector on this line is “lost” or “squashed” to the zero vector by the transformation . This is the information that the matrix loses.
For example, the vector and the vector both get sent to .
3. Row Space, - (The “Active” Inputs)
How to find
The non-zero rows of the RREF () form a basis for the row space.
Basis
Geometric Meaning
The row space is a plane in 3D space.
What it tells us
This plane represents the domain of “effective” inputs. Any part of an input vector that is orthogonal (perpendicular) to this plane will be in the null space and get sent to zero.
Key Connection
Notice that the basis vector for the Null Space is orthogonal to both basis vectors for the Row Space:
This is always true: The Null Space and the Row Space are orthogonal complements.
4. Left Null Space, - (The Constraints on the Outputs)
How to find
This is the null space of . We can find it by finding the dependencies in the rows of during elimination. In our RREF calculation, the last row became all zeros by the operation R3 → R3 + R1 - 2R2 (after a couple of steps). This relationship gives us the basis vector.
A more direct way is to solve .
Basis
Geometric Meaning
This is a line in 3D space.
What it tells us
This space defines the constraints for a vector to be in the column space. A vector is a valid output if and only if it is orthogonal to the left null space.
Key Connection
The basis vector for the Left Null Space is orthogonal to both basis vectors for the Column Space:
This tells us that for to have a solution, must satisfy . This is the equation of the plane we found for the column space!
Fundamental Theorem of Linear Algebra
- (2 + 1 = 3)
- (2 + 1 = 3)
Understanding these subspaces allows us to completely understand the behavior of the linear transformation that the matrix represents.
A subspace must be a “flat” object that passes through the origin.
It can be:
- A point (just the origin)
- A line through the origin
- A plane through the origin
- All of
But it cannot be curved, shifted away from the origin, or have boundaries.
To determine if a set is a subspace of , you must check if it meets three conditions:
The Three Conditions
1. Contains the Zero Vector
The set must contain
2. Closed Under Addition
If you add any two vectors from the set, the result is still in the set.
- If , then
3. Closed Under Scalar Multiplication
If you multiply any vector from the set by a scalar, the result is still in the set.
- If and , then
Examples of Subspaces
Example 1: A Plane Through the Origin
Check 1 (Zero vector):
- Does ? ✓ Yes
Check 2 (Addition):
- If , then and
- For : ✓
Check 3 (Scalar multiplication):
- If and : ✓
Conclusion: This is a subspace (a plane through the origin).
Example 2: A Line Through the Origin
All three conditions are satisfied → This is a subspace.
Examples of Non-Subspaces
Example 1: Plane Not Through Origin
Check 1: Does satisfy ?
- ✗
Conclusion: Not a subspace (fails condition 1).
Example 2: Upper Hemisphere
Check 2: Take and
Both are in , but
Check: ✗
Conclusion: Not a subspace (fails closure under addition).