2. Solving Linear Equations

2.1 Linear Equations Picture

Row Picture

2 by 2 equations

Two equations, Two unknowns

\[\begin{matrix} x - 2y = 1 \\ 3x + 2y = 11 \end{matrix}
\]

The row picture shows two lines meeting at a single point(the solution).

3 by 3 equations

Three equations, Three unknowns

\[Ax=b \ \ => \ \ \begin{matrix} x + 2y + 3z = 6 \\ 2x + 5y + 2z = 4 \\ 6x - 3y + z = 2\end{matrix}
\]

The row picture shows three planes meeting at a single point.

Column Picture

2 by 2 equations

Two equations, Two unknowns

\[\begin{matrix} x - 2y = 1 \\ 3x + 2y = 11 \end{matrix} =>x\left[ \begin{matrix} 1\\ 3 \\ \end{matrix} \right] + y\left[ \begin{matrix} -2\\ 2 \\ \end{matrix} \right] =\left[\begin{matrix} 1 \\ 11 \end{matrix}\right] = b
\]

The column picture combines the column vectors on the left side to produce the vector b on the right side.

(The left side of the vector equation is a linear combination of the columns)

3 by 3 equations

Three equations, Three unknowns

\[\begin{matrix} x + 2y + 3z = 6 \\ 2x + 5y + 2z = 4 \\ 6x - 3y + z = 2 \end{matrix} =>x\left[ \begin{matrix} 1\\ 2\\6 \\ \end{matrix} \right] + y\left[ \begin{matrix} 2\\ 5 \\-3 \\ \end{matrix} \right] + z\left[ \begin{matrix} 3\\ 2 \\1 \\ \end{matrix} \right] =\left[\begin{matrix} 4 \\ 6 \\ 2\end{matrix}\right] = b
\]

The column picture combines three columns to produce b，the coefficients (x,y,z) = (0,0,2).

2.2 Elimination

2.2.1 Gaussian Elimination

• Column 1 : Use the first equation to create zeros below the first pivot.
• Column 2 : Use the new equation 2 to create zeros below the second pivot.
• Column 3 to n : Keep going to find all n pivots and the upper triangular U.

2 by 2

Multiply equation 1 by 3, and Subtract from equation 2.

\[(Before) \ \ \begin{matrix} x - 2y = 1 \\ 3x + 2y = 11 \end{matrix} ==> (After) \ \ \begin{matrix} x - 2y = 1 \\ 8y = 8 \end{matrix}
\]

3 by 3

\[Ax = b \ ==> \ \begin{matrix} 2x + 4y - 2z = 2 \\ 4x + 9y - 3z = 8 \\ -2x-3y+7z=10 \end{matrix}
\]

Elimination Steps

step1 : Subtract 2 times equation 1 from equation 2.

\[\begin{matrix} 2x + 4y - 2z = 2 \\
\quad \quad \quad y + z = 4 \\
-2x-3y+7z=10 \end{matrix}
\]

step2 : Subtract -1 times equation 1 from equation 3.

\[\begin{matrix} 2x + 4y - 2z = 2 \\
\quad \quad \quad y + z = 4 \\
\quad \quad \quad \quad y+5z=12 \end{matrix}
\]

step3 : Subtract new equation 2 from new equation 3.

\[\quad \begin{matrix} 2x + 4y - 2z = 2 \\
\quad \quad \quad \quad y + z = 4 \\
\quad \quad \quad \quad \quad 4z = 8 \end{matrix}
==>Ux = c
\]

U is upper triangular.

Back substitution

z = 2 --> y = 2 --> x = -1

2.2.2 Elimination-Matrix

Elimination multiplies Ax=b by \(E_{21} , E_{31} , E_{41}, ..., E_{n1}\), then \(E_{32} , E_{42}, ..., E_{n2}\) and onward.

• \(E =E_{21} ,..., E_{n1},..., E_{n2},...,E_{n(n-1)}\) , \(EA = [Ea_1...Ea_n]\)
• Augmented matrix : \(E[A\ \ b] = [EA\ \ Eb]\)

example:

\[Ax = b \\
\Downarrow \\
\begin{matrix} 2x_1 + 4x_2 - 2x_3 = 2 \\ 4x_1 + 9x_2 - 3x_3 = 8 \\ -2x_1-3x_2+7x_3=10 \end{matrix} \\
\Downarrow \\
\left[ \begin{matrix} 2&4&-2 \\ 4&9&-3 \\ -2&-3&7 \end{matrix} \right]
\left[ \begin{matrix} x_1\\x_2\\x_3 \end{matrix} \right] =
\left[ \begin{matrix} 2\\8\\10 \end{matrix} \right] \\
\Downarrow \\
\left[ \begin{matrix} 1&0&0 \\ -2&1&0 \\ 0&0&1 \end{matrix} \right]
\left[ \begin{matrix} 2&4&-2 \\ 4&9&-3 \\ -2&-3&7 \end{matrix} \right]
\left[ \begin{matrix} x_1\\x_2\\x_3 \end{matrix} \right] =
\left[ \begin{matrix} 1&0&0 \\ -2&1&0 \\ 0&0&1 \end{matrix} \right]
\left[ \begin{matrix} 2\\8\\10 \end{matrix} \right] \\
\Downarrow \\
\left[ \begin{matrix} 2&4&-2 \\ 0&1&1 \\ -2&-3&7 \end{matrix} \right]
\left[ \begin{matrix} x_1\\x_2\\x_3 \end{matrix} \right] =
\left[ \begin{matrix} 2\\4\\10 \end{matrix} \right] \\
\Downarrow \\
\left[ \begin{matrix} 1&0&0 \\ 0&1&0 \\ 1&0&1 \end{matrix} \right]
\left[ \begin{matrix} 2&4&-2 \\ 0&1&1 \\ -2&-3&7 \end{matrix} \right]
\left[ \begin{matrix} x_1\\x_2\\x_3 \end{matrix} \right] =
\left[ \begin{matrix} 1&0&0 \\ 0&1&0 \\ 1&0&1 \end{matrix} \right]
\left[ \begin{matrix} 2\\4\\10 \end{matrix} \right] \\
\Downarrow \\
\left[ \begin{matrix} 2&4&-2 \\ 0&1&1 \\ 0&1&5 \end{matrix} \right]
\left[ \begin{matrix} x_1\\x_2\\x_3 \end{matrix} \right] =
\left[ \begin{matrix} 2\\4\\12 \end{matrix} \right] \\
\Downarrow \\
\left[ \begin{matrix} 1&0&0 \\ 0&1&0 \\ 0&-1&1 \end{matrix} \right]
\left[ \begin{matrix} 2&4&-2 \\ 0&1&1 \\ 0&1&5 \end{matrix} \right]
\left[ \begin{matrix} x_1\\x_2\\x_3 \end{matrix} \right] =
\left[ \begin{matrix} 1&0&0 \\ 0&1&0 \\ 0&-1&1 \end{matrix} \right]
\left[ \begin{matrix} 2\\4\\12 \end{matrix} \right] \\
\Downarrow \\
\left[ \begin{matrix} 2&4&-2 \\ 0&1&1 \\ 0&0&4 \end{matrix} \right]
\left[ \begin{matrix} x_1\\x_2\\x_3 \end{matrix} \right] =
\left[ \begin{matrix} 2\\4\\8 \end{matrix} \right] \\
\Downarrow Back \ \ substitution \\
x_3 = 2 , x_2 = 2, x_1 = -1
\]

2.3 Rules for Matrix Operations

2.3.1 Matrix Multiplication

Matrices A with n columns multiply matrices B with n rows : \(A_{m \times n} B_{n \times p} = C_{m \times p}\)

The regular way

The entry in row i and column j of AB is (row i of A) \(\cdot\) (column j of B): \((AB)_{ij}=a_{i1}b_{1j} + a_{i2}b_{2j}+...+a_{in}b_{nj}\)

\[\left[ \begin{matrix} * \\ a_{i1}&a_{i2}&...&a_{in} \\ * \\* \end{matrix} \right]
\left[ \begin{matrix} *&b_{1j}&*&*\\ &b_{2j}&&\ \\ &\vdots&& \\ &b_{nj}&& \end{matrix} \right]=
\left[ \begin{matrix} &&*&& \\ *&*&(AB)_{ij}&*&* \\ &&*&& \\&&*&& \end{matrix} \right]
\]

The column way

Each column of AB is a combination of the columns of A.

Matrix A times every column of B : \(A[b_1...b_p]=[Ab_1...Ab_p]\)

The row way

Every row of AB is a combination of the rows of B

Every row of A times matrix B : \(\left[\begin{matrix} a_1 \\ a_2 \\ \vdots \\a_n \end{matrix}\right]B=\left[\begin{matrix} a_1B \\ a_2B \\ \vdots \\a_nB \end{matrix}\right]\)

The columns multiply rows

Multiply columns 1 to n of A times rows 1 to n of B. Add those matrices.

\[\left[\begin{matrix} col_1&\cdots&col_n \\ \vdots&\vdots&\vdots \end{matrix}\right]
\left[\begin{matrix} row_1&\cdots \\ \vdots&\vdots \\row_n&\cdots \end{matrix}\right]
=(col_1)(row_1)+...+(col_n)(row_n)
\]

Block Multiplication

A and B cut into blocks(which are small matrices).

\[A = \left[\begin{matrix} A_1&A_2\\ A_3&A_4 \end{matrix}\right] \\
B = \left[\begin{matrix} B_1&B_2\\ B_3&B_4 \end{matrix}\right] \\
AB =\left[\begin{matrix} A_1&A_2\\ A_3&A_4 \end{matrix}\right]
\left[\begin{matrix} B_1&B_2\\ B_3&B_4 \end{matrix}\right] =
\left[\begin{matrix} A_1B_1 + A_2B_3&A_1B_2 + A_2B_4\\ A_3B_1 + A_4B_3&A_2B_2 + A_4B_4\end{matrix}\right]
\]

2.3.2 The Laws for Matrix Operations

Additions

Commutative law : A + B = B + A

Distributive law : c(A + B) = cA + cB

Associative law : A + (B + C) = (A + B) + C

Multiply

Commutative law is usually broken : \(AB \neq BA\)

Distributive law : (A + B)C = AC + BC or C(A + B) = CA + CB

Associative law : A (B C) = (A B) C

2.4 Inverse Matrices

The matrix A is invertible if there exists a matrix \(A^{-1}\) that "inverts" A :

\[A^{-1}A = I \quad and \quad AA^{-1}=I
\]

• A is invertible if and only if it has n pivots (row exchanges allowed).
• If Ax = 0 for a nonzero vector x, then A has no inverse.
• The inverse of AB is the reverse product \(B^{-1}A^{-1}\)，and \((ABC)^{-1}=C^{-1}B^{-1}A^{-1}\).
• Diagonally dominant matrices are invertible.Each \(|a_{ii}|\)dominates its row.

Gauss-Jordan Method

\[[A \quad I] \quad reduce \quad to \quad [I \quad A^{-1}]
\]

example $A = \left[ \begin{matrix} 2&3 \ 4&7 \end{matrix}\right] $:

\[[A \quad I] = \left[ \begin{matrix} 2&3&1&0 \\ 4&7&0&1 \end{matrix}\right] \\
\Downarrow \\
[U \quad L^{-1}]=\left[ \begin{matrix} 2&3&1&0 \\ 0&1&-2&1 \end{matrix}\right] \quad \\
\Downarrow \\
\left[ \begin{matrix} 2&0&7&-3 \\ 0&1&-2&1 \end{matrix}\right] \\
\Downarrow \\
[I \quad A^{-1}]=\left[ \begin{matrix} 1&0&7/2&-3/2 \\ 0&1&-2&1 \end{matrix}\right] \quad \\
\]

2.5 Factorization : A = LU

Gaussian elimination (with no row exchanges) factors A into L times U，the factors L and U are triangular matrices, and L include all their inverse.

\[A = LU \quad (which \quad L \rightarrow lower \quad triangular, \quad U \rightarrow upper \quad trangular)
\]

\[(E_{n(n-1)}...E_{31}E_{21})A = U \\
\Downarrow \\
(E_{21}^{-1}E_{31}^{-1}...E_{n(n-1)}^{-1})(E_{n(n-1)}...E_{31}E_{21})A = (E_{21}^{-1}E_{31}^{-1}...E_{n(n-1)}^{-1})U \\
\Downarrow \\
A = LU \\
\]

example \(A = \left[ \begin{matrix} 2&1&0 \\ 1&2&1 \\ 0&1&2 \end{matrix}\right] =
\left[ \begin{matrix} 1&0&0 \\ 1/2&1&0 \\ 0&2/3&1 \end{matrix}\right]
\left[ \begin{matrix} 2&1&0 \\ 0&3/2&1 \\ 0&0&4/3 \end{matrix}\right] = LU\)

The triangular factorization can be written : \(A = LU \rightarrow A=LDU\), that D is a diagonal matrix contains the pivots.

Split U into \(DU=\left[ \begin{matrix} d_1&&& \\ &d_2&& \\ &&\ddots \\ &&&d_n \end{matrix}\right]\left[ \begin{matrix} 1&u_{12}/d_1&u_{13}/d_1&\cdots \\ &1&u_{23}/d_2&\vdots \\ &&\ddots \\ &&&1 \end{matrix}\right]\)

example:

\[A = \left[ \begin{matrix} 2&1&0 \\ 1&2&1 \\ 0&1&2 \end{matrix}\right] \\ =
\left[ \begin{matrix} 1&0&0 \\ 1/2&1&0 \\ 0&2/3&1 \end{matrix}\right]
\left[ \begin{matrix} 2&1&0 \\ 0&3/2&1 \\ 0&0&4/3 \end{matrix}\right] \\ =
\left[ \begin{matrix} 1&0&0 \\ 1/2&1&0 \\ 0&2/3&1 \end{matrix}\right]
\left[ \begin{matrix} 2&0&0 \\ 0&3/2&0 \\ 0&0&4/3 \end{matrix}\right]\left[ \begin{matrix} 1&1/2&0 \\ 0&1&2/3 \\ 0&0&1 \end{matrix}\right]= LDU
\]

Keys

• The lower triangular L contains the number \(l_{ij}\) that multiply pivot rows, going from A to U. The product LU adds those rows back to recover A.
• On the right side we solve Lc = b (forward) and Ux=c (backward).
• Cost : the left side costs \(1/3(n^3 -n)\) multiplications and subtractions，the right side costs \(n^2\) multiplications and subtractions.

2.6 Transposes and Permutations

Transposes

The columns of \(A^{T}\) are the rows of A

\[(A^{T})_{ij} = A_{ji}
\]

If \(A = \left [ \begin{matrix} 1&2&3 \\ 0&0&4 \end{matrix}\right]\) then \(A^{T} = \left [ \begin{matrix} 1&0 \\ 2&0 \\ 3&4 \end{matrix}\right]\)

Sum : \((A+B)^{T} = A^{T} + B^{T}\)

Product : \((AB)^{T} = B^{T}A^{T}\)

Inverse : \((A^{T})^{-1} = (A^{-1})^{T}\)

Symmetric matrix (\(S^T=S\))：\(U = L^T \rightarrow S = LDU = LDL^T\)

Permutations

A permutation matrix P has the rows of the identity I in any order, \(P_{ij}\) is constructed by exchanging two row i and j of \(I\)，and there are \(n!\) permuataion matrices of order n.

3 by 3 permuation matrices:

\[I = \left [ \begin{matrix} 1&& \\ &1& \\ &&1 \end{matrix}\right] \quad
P_{21} = \left [ \begin{matrix} &1& \\ 1&& \\ &&1 \end{matrix}\right] \quad
P_{31} = \left [ \begin{matrix} &&1\\ &1& \\ 1&& \end{matrix}\right] \\
P_{32} = \left [ \begin{matrix} 1&&\\ &&1 \\ &1& \end{matrix}\right] \quad
P_{32}P_{21} = \left [ \begin{matrix} &1&\\ &&1 \\ 1&& \end{matrix}\right] \quad
P_{21}P_{32} = \left [ \begin{matrix} &&1\\ 1&& \\ &1& \end{matrix}\right]
\]

• If A is invertible then a permutation P will reorder its rows for PA=LU.
• A permutation matrix P has a 1 in each row and column, and \(P^T = P^{-1}\).