Linear algebra A brush-up course Anders Ringgaard Kristensen Presented by Dan Jensen
Linear algebraA brush-up course
Anders Ringgaard KristensenPresented by Dan Jensen
Outline
Real numbers
• Operations
• Linear equations
Matrices and vectors
Systems of linear equations
Let us start with something familiar!
Real numbers!
The real number system consists of 4 parts:• A set R of all real numbers
• A relation < on R. If a, b ⊆ R, then a < b is either true or false. It is called the order relation.
• A function +: R + R �R . The addition operation
• A function ∘ : R ∘ R � R . The multiplication operation.
A number of axioms apply to real numbers
Axioms for real numbers I
Associative laws• a + (b + c) = (a + b) + c
• a ∘ (b ∘ c) = (a ∘ b) ∘ c
Commutative laws• a + b = b + a
• a ∘ b = b ∘ a
Distributive law• a ∘ (b + c) = a ∘ b + a ∘ c
Axioms for real numbers II
Additive identity (”zero” element)• There exist an element in R called 0 so that, for all a, a + 0 = a
Additive inverse• For all a there exists a b so that a + b = 0, and b = − a
Multiplicative identity (”one” element)• There exists an element in R called 1 so that, for all a, 1 ∘ a = a
Multiplicative inverse• For all a ≠ 0 there exists a b so that a ∘ b = 1, and b = a-1
Solving equations
Let a ≠ 0 and b be known real numbers, and x be an unknown real number.
If, for some reason, we know that a ∘ x = b, we say that we have an equation.
We can solve the equation in a couple of stages using the axioms:
a ∘ x = b ,a-1 ∘ a ∘ x = a-1 ∘ b ,1 ∘ x = a-1 ∘ b ,x = a-1 ∘ b
Example of a trivial equation
Farmer Hansen has delivered 10000 kg milk to the dairy last week. He received a total payment of 23000 DKK. From this information, we can find the milk price per kg
(a = 10000, b = 23000, x = milk price):
• 10000 ∘ x = 23000 ,• x = 10000-1 ∘ 23000 = 0.0001 ∘ 23000 = 2.30
So, the milk price is 2.30 DKK/kg
a ∘ x = b , x = a-1 ∘ b
What is a matrix? Examples:
A 2 x 3 matrix:
A 4 x 3 matrix:
Symbol notation for a 2 x 2 matrix:
Special matrices
A matrix a of dimension n x n is called a quadratic matrix:
A matrix b of dimension 1 x n is called a row vector:
A matrix c of dimension n x 1 is called a column vector:
Operations: Addition
Two matrices a and b may be added, if they are of same dimension (say n x m):
From the axioms of real numbers, it follows directly that the commutative law is also valid for matrix addition:• a + b = b + a
Class Question: Additive identity?
Does the set of n x m matrices have a ”zero”
element 0 so that for any a, a + 0 = a
If yes, what does it look like?
Additive inverse
It follows directly from the axioms for real numbers, that every matrix a, has an additive inverse, b, so that a + b = 0 , and, for the additive inverse, b = −a
2-5 minute break
Operations: Multiplication
Two matrices a and b may be multiplied, if a is of dimension n x m, and b is of dimension m x k
The result is a matrix of dimension n x k .
Due to the dimension requirements, it is clear that the commutative law is not valid for matrix multiplication. • Even when b ∘ a exists, most often a ∘ b ≠ b ∘ a
Matrix multiplication – simple example
An element in the product is
calculated as the product of a row and a column
5 4
3 6
1 2
2 3 2
1 2 4
3 2 1
21 30
15 24
22 26
A 3 x 3 matrix multiplied with a 3 x 2 matrix
Vector multiplication
A row vector a of dimension 1 x n may be
multiplied with a column vector b of dimension n x 1. The product a ∘ b is a 1 x1 matrix (i.e. a real number), whereas the product b ∘ a is a quadratic n x n matrix:
Class Question: Multiplicative identity
Does the set of matrices have a ”one” elementI1, so that if I1 is an n x m matrix, then for anym x k matrix a, I1∘ a = a
If yes:• What must the value of n necessarily be?• What are the elements of I1 – what does the matrix
look like?
Does there exist a ”one” element I2 so that forany matrix a of given dimension, a ∘ I2 = a
If yes: Same questions as before
Other matrix operations
A real number r may be multiplied with a matrix a
The transpose a’ of a matrix a is formed by changing columns to rows and vice versa:
Other matrix operations: Examples
If r = 2, and then:
The transpose a’ of a is
Class Question: Multiplicative inverse I
Does every matrix a ≠ 0 have a multiplicative inverse, b, so that a ∘ b = I
If yes,
• What does it look like?
Multiplicative inverse II
A matrix a only has a multiplicative inverse under certain conditions:• The matrix a is quadratic (i.e. the
dimension is n x n)
• The matrix a is non-singular:
• A matrix a is singular if and only if
det(a) = 0,
where det(a) is the determinant of a
• For a quadratic zero matrix 0, we have det(0) = 0, so 0 is singular (as expected)
• Many other quadratic matrices are singular as well �
Determinant
The determinant of a quadratic matrix is a real number.
Calculation of the determinant is rather complicated for large dimensions.
The determinant of a 2 x 2 matrix:
The determinant of a 3 x 3 matrix:
The (multiplicative) inverse matrix
If a quadratic matrix a is non-singular, it has an inverse a-1, and: • a ∘ a-1 = I • a-1 ∘ a = I
The inverse is complicated to find for matrices of high dimension.
For real big matrices (millions of rows and columns) inversion is a challenge even to modern computers.
Inversion of matrices is crucial in many applications in herd management (and animal breeding)
Inversion of ”small” matrices I
A 2 x 2 matrix a is inverted as
Example
Inversion of ”small” matrices II
A 3 x 3 matrix a is inverted as
Example
2-5 minute break
Why do we need matrices?
Because they enable us to express very complexrelations in a very compact way.
Because the algebra and notation are powerful tools in mathematical proofs for correctnessof methods and properties.
Because they enable us to solve large systems of linear equations.
Complex relations I
Modeling of drinking patterns of weaned piglets.
Complex relations
Madsen et al. (2005) performed an on-line monitoring of the water intake of piglets. The water intake Yt at time twas expressed as
Where
Simple, but …
Complex relations II
F, θt and wt are of dimension 25 x 1, G and Wt are of dimension 25 x 25.
The value of θt is what we try to estimate.
Systems of linear equations
A naïve example: Old McDonald has a farm …On his farm he has some sheep, but he has forgotten how
many. Let us denote the number as x1 .On his farm he has some geese, but he has forgotten how
many. Let us denote the number as x2 .He has no other animals, and the other day he counted the
number of heads of his animals. The number was 25. He knows that sheep and geese have one head each, so he set up the following equation:• 1x1 + 1x2 = 25
He also counted the number of legs, and it was 70. He knows that a sheep has 4 legs and a goose has 2 legs, so he set up the following equation:• 4x1 + 2x2 = 70
Old McDonald’s animals
We have two equations• 1x1 + 1x2 = 25
• 4x1 + 2x2 = 70
Define the following matrix a and the (column-) vectors x and b
We may then express the two equations as one matrix equation:
Solving systems of linear equations
Having brought the system of linear equations to the elegant form, solution for x is just as straight forward as with an equation of real numbers:
This is true no matter whether we have a system of 2 equations like here, or we have a system of a million equations (which is not at all unrealistic).
a ∘ x = b ,a-1 ∘ a ∘ x = a-1 ∘ b ,I ∘ x = a-1 ∘ b ,x = a-1 ∘ b
Linear regression and matrices I
In a study of children born in Berkeley 1928-29 the height and weight of 10 18-year old girls were measured.
It is reasonable to assume that the weight Yi depends on the height xi according to the following linear regression model:• Yi = β0 + β1xi + εi where,
� β0 and β1 are unknown parameters
• The εi are N(0, σ2)
Linear regression and matrices II
Let us define the following matrices:
We may then write our model in matrix notation simply as:
• Y = xβ + ε
Linear regression and matrices III
The least squares estimate of β is
Define the vector of predictions as
Then an estimate s2 for σ2 is
• Where n = 10 is the number of observations and k = 2 is the number of parameters estimated.
Applying these formulas yields:
Linear regression and matrices IV
Visual inspection of the fitted curve
Weight versus heigth of 18-year old girls
4550556065707580
150 160 170 180 190
Heigth, cm
Wei
gh
t, k
g
Observations Fitted regression
A class variable: Boys and girls I
If it had been 5 girls and 5 boys we had observed, the data could have looked like this (where xi1 = 0 means girl and xi1 = 1 means boy):
A class variable: Boys and girls II
We obtain the following estimate for β
The interpretation is that the weight of a boy is 4.49 kg lower than the weight of a girl of exactly same height.
(Since we have declared 5 arbitrarily selected girls for boys, the result should not be interpreted at all)
R and R studio for statistical computing:
R and R studio for statistical computing:
R R Studio
Independant program GUI for R
Command line basedCommand line
AND”Point and Click”
Hard to keep an overview –one thing at a time
Easy to keep an overvirew –multible tabs and windsows
Mac and Windows Mac and Windows
www.r-project.org/ www.rstudio.com/products/rstudio/
Matrix operations in R:
Make a 2x3 matrix:
A = matrix(c(1, 2, 3, 4, 5, 6), 2,3)
Add matricies A and B:
A+B
Multiply number and matrix:
2*A
Multiply matrices A and B:
A%*%B
Transpose matrix A:
t(A)
Invers square matrix A:
solve(A)