Clever Algorithms Welcome to CleverAlgorithms.com! Get notified of future announcements! Email:

 Clever Algorithms: Statistical Machine Learning Recipes By Jason Brownlee PhD. First Edition, Lulu Enterprises, [Expected mid 2012]. ISBN: XXX.

Please Note that this is an early access preview of the book.
If you find a typo or have any suggestions, please send an email to Jason: jasonb@CleverAlgorithms.com.

Get notified when the book is released!
Email:

# Ordinary Least Squares Regression

Ordinary Least Squares Regression, Ordinary Linear Regression, OLS, OLSR, Linear Least Squares

## Taxonomy

Ordinary Least Squares is a method for estimating the parameters for a regression model. It is related to other least squares methods for estimating the parameters of linear models such as Weighted Least Squares and Partial Least Squares.

## Strategy

The information processing objective of the Ordinary Least Squares Regression is to define a line a best fit. The line is defined by an equation that minimizes the Sum of the Squared Residuals (SSR) with an intercept and a regression coefficient.

## Heuristics

• An OLS model is asymptotically consistent when 1) the errors are unbiased (zero mean) and 2) the coefficients (regressors) are linearly independent.
• The Gauss-Markov theorem indicates that a linear regression model where 1) all errors have the same variance (hoemoscedasticity), and 2) all errors are drawn from uncorrelated distributions (nonautocorrelation) then the coefficients given by OLS result in the Best Linear Unbiased Estimator (BLUE).
• Under the Gauss-Markov conditions, if the errors are further 1) independent and identically distributed (iid), and 2) drawn from a normal distributions then the models estimates may be taken as maximum likelihood estimates.
• The model assumes attributes are drawn from a normal distribution and a straight line can be drawn through the mean (under the normal and iid assumptions).
• The dependent variable must be continuous, but the independent variables may be continuous or categorical.
• Relative to modern methods the prediction accuracy is poor and interpretation of the model can be difficult.
• An $R^2$ statistic can be used to indicate the "goodness of fit" of a resulting regression line. It represents the proportion of the variance of the dependent variable that can be explained by the independent variables. An $R^2=1$ is a perfect fit, and an $R^2=0$ means that the coefficients cannot explain any observations.
• The magnitude of a coefficient indicates the relative importance of its independent variable to the model.

## Code Listing

Listing (below) provides a code listing of the Ordinary Least Squares method in R. Figure (below) provides a plot of the training dataset with the line of best fit highlighted.

The example uses the lm function in the stats core package which is responsible for fitting linear least squares models [RDCT2011a]. For more information on this library type: library(help="stats"), and for more information on the function type: ?lm.

The test problem is a two-dimensional dataset of 100 samples, where the x-values are drawn from a uniformly random distribution $x \in [0,10]$ and y values are the x value plus a value drawn from a normally random distribution with a mean of 0 and a standard deviation of 1.

# define 2D regression problem of 100 samples
regression_dataset <- function() {
x <- runif(100, 0, 10)
y <- x + rnorm(100)
data.frame(x, y)
}

# get the data
data <- regression_dataset()
# split data in to train and test (67%/33%)
training_set <- sample(100,67)
train <- data[training_set,]
test <- data[(1:100)[-training_set],]

# create a linear regression model using ordinary least squares
model <- lm(
y~x, # predict Y given X
train, # training dataset
NULL, # no weighting on the variables
NULL, # no action on missing values
method="qr") # QR decomposition (efficient matrix calculation method)

# summarize the create linear regression model
summary(model)
# regression model diagnostic plots with the training data
par(mfrow=c(2,2))
plot(model)

# plot the training data and the line of best fit
plot(train)
abline(model$coef, lty=5, col="red") # make predictions for the test data predictions <- predict.lm(model, test[,1:2]) # compute mean squared error print(mean((test$y-predictions)^2))
Example of Ordinary Least Squares in R using the lm function in the stats core package.
Download: stats_ordinary_least_squares.R. Unit test available in the github project

Plot 2D training dataset with the line of best fit.

## References

### Primary Sources

The method for Least Squares may be traced back to Gauss who claims to have devised the method in and later published it in 1809 [Gauss1809] [Gauss1823] (Latin). Legendre independently developed and published the Least Squares method in 1805 [Legendre1805] (Appendix, French).

### More Information

Least Squares Regression is covered in any good text of statistics, data analysis or econometrics. Some popular introductory texts to least squares regression include Montgomery et al. [Montgomery2001] (Chapter 2), and Tamhane and Dunlop [Tamhane2000] (Chapter 10). Faraway provides a free book that includes a practical introduction into Least Squares Regression with examples in R [Faraway2002] (updated version published [Faraway2004]).

## Bibliography

 [Faraway2002] J. J. Faraway, "Practical Regression and Anova in R", University of Michigan, 2002. [Faraway2004] J. Faraway, "Linear Models with R", Chapman and Hall/CRC, 2004. [Gauss1809] C. F. Gauss, "Theoria motus corporum coelestium in sectionibus conicis solem ambientium", 1809. [Gauss1823] C. F. Gauss, "Theoria combinationis observationum erroribus minimis obnoxiae", H. Dieterich, 1823. [Legendre1805] A. M. Legendre, "Nouvelles méthodes pour la détermination des orbites des cometes", F. Didot, 1805. [Montgomery2001] D. C. Montgomery and E. A. Peck and G. G. Vining, "Introduction to Linear Regression Analysis", Wiley-Interscience, 2001. [RDCT2011a] R Development Core Team, "Package 'stats': The R Stats Package", 2011. [Tamhane2000] A. C. Tamhane and D. D. Dunlop, "Statistics and data analysis: from elementary to intermediate", Prentice Hall, 2000.

Coming soon...

Coming soon...

### More in the Series

Check-out other books in the series.

Please Note: This content was automatically generated from the book content and may contain minor differences.