Least Squares Regression without Matrices
April 16th, 2008Recently I was trying to figure out a way to do Least Square Regression without having to use matrices. I looked for quite awhile online for some ideas on how I could do this. Using Matrices to tackle regression problems become a problem themselves because the matrix computations required to determine regression take a ton of computer power and often bog down systems. It is just too much for the computer to do quickly. So a way around this is by using summations.
With Least Squares Regression we are trying to determine our intercept and our coefficients of x. You do have to be familiar with summation notation to understand this. What we are basically doing in this approach is trying to minimize the difference between are dependent variable, y, and our function of x, f(x). The summation notation for this is:
∑[y-f(x)]^2=minimum
Because our f(x) is essentially f(x)=y=a+bx, so then we can have a summation of:
∏=∑[y-(a+bx)]^2=minimum
In order to minimize this summation we must take the partials derivatives of ∏ with respect to a and b.
∂∏/∂a = 2∑[y-(a+bx)]=0;
∂∏/∂b = 2∑x[y-(a+bx)]=0
From this we can derive summation equations to obtain the unknown coefficients of a and b:
a={(∑y)(∑[x^2])-(∑x)(∑xy)} / {n(∑[x^2])-[(∑x)^2]}
b= {n∑xy-(∑x)(∑y)} / {n(∑x^2)-[(∑x)^2]}
This is all we need to determine the coeffficients for Least Squares Regression. It is an easy solution to avoid the headaches of the complexity of matrix computations.