and b is in 
, then a
least-squares solution of 
is 
in 
such
that
Least-Squares Problems
Summary
Okay! This is it: the section with the formula for the solution of the least-squares problem, which is known as the linear regression problem in statistics. This is how we find a nice fit to linear (and specialized types of non-linear) models. What an amazingly powerful tool this is, and it's based on some simple linear algebra....
least-squares solution: If and b is in , then a
least-squares solution of is in such
that
for all x in .
Q: Take a look at that equation above, and tell me where the name ``least-squares'' comes from....
Now, consider the projection of b onto the Col A,
and let be defined as the solution of
Q: How do we know that there is such a solution?
We know that 
is orthogonal to Col A, so
from which we arrive at
Hence is a solution of the equation
(the so-called normal equations). There may be many (in infinite number!) of solutions of the normal equations.
Theorem 13: The set of least-squares solutions of
coincides with the nonempty set of solutions of the normal equations
.
However, if 
is invertible, then the solution is unique:
Theorem 14: The matrix is invertible 
the columns of A
are linearly independent. In this case, the equation has
only one least-squares solution , and it is
Problems: