3. Vector Spaces and Subspaces

3.1 Vector Spaces

The space $R^n$ consists of all colunm vectors $v$ with n components. We can add any vectors in $R^n$ , and we can multiply any vector $v$ by any scalar c , the result stays in the space $R^n$.

examples:

columns between brackets : $\left[ \begin{matrix} 4 \\ \pi \end{matrix} \right]$ is in $R^2$

commas and parentheses : (1,1,0,1,1) is in $R^5$

complex numbers spaces : $\left[ \begin{matrix} 1 + i \\ 1-i \end{matrix} \right]$

3.2 Subspaces

A subspace of a vector space is a set of vectors (including 0) that satisfies two requirements.

If $v$ and $w$ are vectors in the subspace and c is any scalar, then:

rule 1 : $v + w$ is in the subspace.
rule 2 : $cv$ is in the subspace.
rule 1 + rule 2 : $cv + dw $ is in the subspace. (subspace is closed.)

Every subspace contains the zero vector.

Example: $R^3$ subspaces

The whole space $R^3$ is a subspace (of itself)-- The largest one.
Any lines through (0,0,0) -- L
Any plane through (0,0,0) -- P
The single vector (0,0,0) -- Z (zero vectors space--The littlest one)

3.3 Column Space

Ax =b , the column space of A is denoted by C(A) , C(A) contains not just the n columns of A, but all their combinations Ax，C(A) is a subspace of $R^m$.

The system Ax=b is solvable if and only if b is in the column space of A.

\[C(A) = all \quad combinations \quad of \quad the \quad columns = all \quad vectors \quad Ax
\]

example:

\[Ax = \left[ \begin{matrix} 1&0 \\ 4&3 \\ 2&3 \end{matrix}\right]
\left[ \begin{matrix} x_1 \\ x_2 \end{matrix}\right] =
x_1\left[ \begin{matrix} 1 \\ 4 \\ 2 \end{matrix}\right] +
x_2\left[ \begin{matrix} 0 \\ 3 \\ 3 \end{matrix}\right]
\]

3.4 The Nullspace : Ax=0 or Rx=0

Key notes:

The nullspace N(A) in $R^n$ contains all solutions x to $Ax=0$. This includes x=0.
Elimination (from A to U to R) does not change the nullspace : N(A)=N(U)=N(R).
The reduced row echelon form R=rref(A) has all pivots = 1, with zeros above and below.
If column j of R is free (no pivot), there is "special solution" to Ax=0 with $x_j=1$.
Number of pivots = number of nonzero rows in R = rank r . There are n-r free columns(variables\dimensions).
Every matrix with m<n has nonzero solutions to Ax=0 in its nullspace.

Solution steps:

reducing A to its row echelon form R.
finding the special solutions to Ax=0. ( Ux = 0 or Rx = 0)
N(A) = N(U) = N(R) = all combinations of special solutions.

\[A = \left [ \begin{matrix} p&p&f&p&f \\ \vdots&\vdots&\vdots&\vdots&\vdots \end{matrix}\right] \\
(3 \ \ pivot \ \ columns \ \ p ，2 \ \ free \ \ columns \ \ f) \\
step \ \ 1 \Downarrow \\
R = \left [ \begin{matrix} 1&0&a&0&c \\ 0&1&b&0&d \\ 0&0&0&1&e \\ 0&0&0&0&0 \end{matrix}\right] \\
(I \ \ in \ \ pivot \ \ columns ，F \ \ in \ \ free \ \ columns, 3 \ \ pivots: rank=3 ) \\
step \ \ 2 \Downarrow \\
s_1 =\left[ \begin{matrix} -a \\ -b \\ 1 \\ 0 \\ 0 \end{matrix}\right] \ \
s_2 =\left[ \begin{matrix} -c \\ -d \\ 0 \\ -e \\ 1 \end{matrix}\right] \\
step \ \ 3 \Downarrow \\
N(A) = N(U) = N(R) = all \ \ combinations \ \ of \ \ s_1 \ \ and \ \ s_2.
\]

3.5 The Complete solutions : Ax=b or Rx = d

Key notes:

Complete solution to Ax=b : x = (one particular solution $x_p$) + (any $x_n$ in the nullspace).
Elimination on [A b] leads to [R d]. Then Ax=b is equivalent to Rx=d.
Ax = b and Rx=d are solvable only when all zero rows of R have zeros in d.
When Rx=d is solvable, one very particular solution $x_p$ has all free variables equal to zero.
A has full rank r=n when its nullspace N(A) = zero vector : no free variables.
A has full row rank r = m when its column space C(A) is $R^m$ : Ax=b is always solvable.

Solution steps:

Produce the augmented matrix [A b]
Get Elimination form [R d]
Set free variables = 0, and get a particular solution -- $x_p$
Set free variables = 1 or 0，and get n - r special solutions -- $x_n$
Complete solution: $x = x_p + x_n$

example:

\[Ax = b \\

\Downarrow \\
\left [ \begin{matrix} 1&3&0&2 \\ 0&0&1&4 \\ 1&3&1&6 \end{matrix}\right]
\left [ \begin{matrix} x_1 \\ x_2 \\ x_3 \\ x_4 \end{matrix}\right]
= \left [ \begin{matrix} 1 \\ 6 \\ 7 \end{matrix}\right] \\

\Downarrow \\
[A \ \ b] = \left [ \begin{matrix} 1&3&0&2&1 \\ 0&0&1&4&6 \\ 1&3&1&6&7 \end{matrix}\right] \\
\Downarrow \\
[R \ \ d] = \left [ \begin{matrix} 1&3&0&2&1 \\ 0&0&1&4&6 \\ 0&0&0&0&0 \end{matrix}\right] \\

\Downarrow \\
Pivot \ \ variables(x_1, x_3) = 1, 6 \\
Free \ \ variables (x_2, x_4) = 0, 0 \\
Solution \ \ x_p = (1, 0, 6, 0) \\

\Downarrow \\
Free \ \ variables (x_2, x_4) = 1 \ \ or \ \ 0 \\
s_1 = [-3, 1, 0, 0] \\
s_2 = [-2, 0, -4, 1] \\
x_n =c\left [ \begin{matrix} -3 \\ 1 \\ 0 \\ 0 \end{matrix}\right] + d\left [ \begin{matrix} -2 \\ 0 \\ -4 \\ 1 \end{matrix}\right] \\

\Downarrow \\
x = x_p + x_n = \left [ \begin{matrix} 1 \\ 0 \\ 6 \\ 0 \end{matrix}\right] +
c\left [ \begin{matrix} -3 \\ 1 \\ 0 \\ 0 \end{matrix}\right] +
d\left [ \begin{matrix} -2 \\ 0 \\ -4 \\ 1 \end{matrix}\right]
\]

The four possibilities for linear equations depend on the rank k :

r = m = n , square and invertible, $R = [I]$，Ax = b has 1 solution (unkowns=equations)
r = m < n, short and wide, $R = [I \ \ F]$，Ax = b has infinite solutions (unkowns > equations)
r = n < m，Tall and thin，$R = \left [ \begin{matrix} I \\ F\end{matrix}\right]$, Ax = b 0 or 1 solutions (unkonwns < equations)
r < n and r < m，not full rank，$R = \left [ \begin{matrix} I&F \\ 0&0 \end{matrix}\right]$, Ax=b has 0 or infinite solutions

3.6 Independence, Basis and Dimension

Independent

Independent columns of A : The only solution to Ax=0 is x=0, the nullspace is Z.
Independent vectors : The only zero combination $c_1v_1 + c_2v_2 + ... + c_kv_k = 0$ has all c's = 0.
A matrix with m<n has dependent columns : At least n-m free variables/special solutions.

Basis

A basis for a vector space is a sequence of vectors with two properties:

The basis vectors are linearly independent.
They span the space.
- The rows are in $R^n$ spanning the row space.
- The columns are in $R^m$ spanning the column space.

Every vector $v$ in the space is a combination of the basis vectors, because they span the space, and there is only one way to write $v$ as a combination of the basis vectors.

The pivot columns of A are a basis for its column space. The pivot rows of A are a basis for its row space. So are the pivot rows of its echelon form R.

a basis of $C(A)$ = pivots columns of A.
a basis of $C(A^T)$ = pivots rows of A.

example:

\[A = \left[ \begin{matrix} 2&4 \\ 3&6 \end{matrix}\right]
\quad reduces
\quad to
\quad
R = \left[ \begin{matrix} 1&2 \\ 0&0 \end{matrix}\right]
\]

The column spaces of A and R are different and their bases are different.

Column 1 of R is the pivot column, which alone is a basis for C(R): $\left[ \begin{matrix} 1 \\ 0\end{matrix} \right]$

Column 1 of A is the pivot column, which alone is a basis for C(A): $\left[ \begin{matrix} 2 \\ 3\end{matrix} \right]$

Row 1 of R is the pivot row, which alone is a basis for $C(R^T)=C(A^T)$: $\left[ \begin{matrix} 1&2 \end{matrix} \right]$

Dimension

The dimension of a space is the number of vectors in every basis.

Bases for Matrix Spaces and Function Spaces

Matrix spaces:

The vector space M contains all n by n matrices, its dimension is $n^2$.
The dimension of the subspace of upper triangular matrices is $1/2n^2 + 1/2 n$.
The dimension of the subspace of diagonal matrices is n.
The dimension of the subspace of symmetric matrices if $1/2n^2 + 1/2 n$.

example:

One basis

\[A_1, A_2,A_3,A_4 = \left[ \begin{matrix} 1&0 \\ 0&0 \end{matrix}\right],
\left[ \begin{matrix} 0&1 \\ 0&0 \end{matrix}\right],
\left[ \begin{matrix} 0&0 \\ 1&0 \end{matrix}\right],
\left[ \begin{matrix} 0&0 \\ 0&1 \end{matrix}\right]
\]

Every A combines the basis matrices

\[A = c_1A_1 + c_2A_2 +c_3A_3 +c_4A_4 = \left[ \begin{matrix} c_1&c_2 \\ c_3&c_4 \end{matrix}\right]
\]

Function spaces:

$y''=0$ is solved by any linear function $y = cx + d$，the space bases has x and 1.
$y'' = -y$ is solved by any combination $y= csinx + dcosx$, the space bases has $sinx$ and $cosx$.
$y'' = y$ is solved by any combination $y=ce^x + de^{-x}$，the space bases has $e^x$ and $e^{-x}$.

3.7 Dimensions of the Four Subspaces

Keys notes:

The column space is $C(A)$，a subspace of $R^m$, has a dimensions r，$r=pivot \ \ columns$
The row space is $C(A^T)$，a subspace of $R^n$, has a dimensions r, $r=pivot \ \ rows$
The nullspace is $N(A)$，a subspace of $R^n$, has a dimensions $n-r$, numbers of free variables
The left nullspace is $N(A^T)$，a subspace of $R^m$, has a dimensions $m-r$, numbers of free rows

example : Ax = b

\[A = \left[ \begin{matrix} -1&1&0&0 \\ -1&0&1&0 \\ 0&-1&1&0 \\ 0&-1&0&1 \\ 0&0&-1&1 \end{matrix}\right]
\quad reduces \ \ to \ \
R = \left[ \begin{matrix} 1&0&0&-1 \\ 0&1&0&-1 \\ 0&0&1&-1 \\ 0&0&0&0 \\ 0&0&0&0 \end{matrix}\right]
\]

The column space $C(A)$ : r = 3 independent columns, a basis is the columns 1,2,3 of A.
The row space $C(A^T)$ : r = 3 independent rows, a basis is the rows 1,2,4 of A or R. ($P_{34}EA=R$)
The nullspace N(A) : Set b = 0, N(A) = N(R)，n - r = 4 - 3 = 1 free variables, x = (1, 1, 1, 1) is the basis for N(A).
The nullspace $N(A^T)$ :
- m-r = 5 - 3 = 2 dimensions;
- Solve $A^Ty = 0$，combinations of the rows give zero, get special solutions $y_1 = (1, -1, 1, 0, 0), y_2 = (0,0,-1,1,-1)$，a basis of $N(A^T)$ is $y_1, y_2$.
- If EA=R， the last m-r rows of R are a basis for the left nullspace of A.

3.8 Rank One Matrices

Every rank one matrix is one column times one row : $A=uv^T$

\[\left[ \begin{matrix} 2&3&7&8 \\ 2a&3a&7a&8a \\ 2b&3b&7b&8b \end{matrix} \right]
= \left[ \begin{matrix} 1 \\ a \\ b \end{matrix} \right] \left[ \begin{matrix} 2&3&7&8 \end{matrix} \right]
= uv^T
\]
Every rank r matrix is a sum of r rank one matrices.

\[A =\left[ \begin{matrix} &&& \\ u_1&u_2&u_3 \\ &&& \end{matrix} \right]
\left[ \begin{matrix} v_1^T \\ v_2^T \\ zero \ \ row \end{matrix} \right]
= u_1v_1^T + u_2v_2^T\\

rank_A = rank1 + rank1
\]