Deriving the regression parameters for simple linear regression

Andy Long
Spring, 2018

Clear["Global`*"] (* This removes any definitions that may be in place already; like if you've already been doing some Mathematica, and talking about other functions named f, x-values named a and b, etc. *)

To illustrate the derivation of the regression parameters that minimize the squares of the residuals, we're going to be using the Keeling data from Stewart, p. 25. 

We've already seen how we can find the regression line using Mathematica's  LinearModelFit.

years={0,2,4,6,8,10,12,14,16,18,20,22,24,26,28}
co2={338.7,341.2,344.4,347.2,351.5,354.2,356.3,358.6,362.4,366.5,369.4,373.2,377.5,381.9,385.6}
n=Length[co2]

We want to find a linear function to fit the co2 values: we need to find the parameters a and b:

f[t_]:=a+b*t
(* we can compute the derivatives of f with respect to a or b: *)
D[f[t],a]
D[f[t],b]

Here is the Sum that we want to minimize, call it S. S is a function of a and b, through the function f:

S=Sum[(co2[[i]]-f[years[[i]]])^2,{i,1,n}]

Here we illustrate the partial derivatives of this sum:

Sa=D[S,a]
Sb=D[S,b]

Now we find the values of a and b that make both partial derivatives 0:

Solve[Sa==0 && Sb==0,{a,b}]

We can check the second derivatives, to make sure that they’re positive -- and hence that we have a minimum (but heh, it’s a paraboloid in the form of a bowl -- we know that it’s a minimum!).

Saa=D[Sa,a]
Sbb=D[Sb,b]