Newton's Method

We are going to generate a variation of non-linear regression which is essentially a multivariate form of Newton's method, so we begin there. The idea behind Newton's method is an important one: we attempt to solve a non-linear problem by successive linear approximations. That is, we are going to solve a linear problem to attempt to approach the solution of the non-linear problem.

Newton's method is specifically a procedure designed to iteratively approach the root of a non-linear function. So, for example, we can attempt to find the root of a function f(x) by

1. Starting with a good guess, and
2. Iteratively improving that guess.

So how does it work? We use the linearization of f about our initial guess :

Set y=0, and solve for x:

This is an iterative scheme for successive improvement of our initial guess. It might converge to a true solution of the non-linear problem, which is our hope.