Ordinary Least Squares Regression

Ordinary Least Squares Regression, Ordinary Linear Regression, OLS, OLSR, Linear Least Squares

Taxonomy

Ordinary Least Squares is a method for estimating the parameters for a regression model. It is related to other least squares methods for estimating the parameters of linear models such as Weighted Least Squares and Partial Least Squares.

Strategy

The information processing objective of the Ordinary Least Squares Regression is to define a line a best fit. The line is defined by an equation that minimizes the Sum of the Squared Residuals (SSR) with an intercept and a regression coefficient.

Heuristics

• An OLS model is asymptotically consistent when 1) the errors are unbiased (zero mean) and 2) the coefficients (regressors) are linearly independent.
• The Gauss-Markov theorem indicates that a linear regression model where 1) all errors have the same variance (hoemoscedasticity), and 2) all errors are drawn from uncorrelated distributions (nonautocorrelation) then the coefficients given by OLS result in the Best Linear Unbiased Estimator (BLUE).
• Under the Gauss-Markov conditions, if the errors are further 1) independent and identically distributed (iid), and 2) drawn from a normal distributions then the models estimates may be taken as maximum likelihood estimates.
• The model assumes attributes are drawn from a normal distribution and a straight line can be drawn through the mean (under the normal and iid assumptions).
• The dependent variable must be continuous, but the independent variables may be continuous or categorical.
• Relative to modern methods the prediction accuracy is poor and interpretation of the model can be difficult.
• An $R^2$ statistic can be used to indicate the "goodness of fit" of a resulting regression line. It represents the proportion of the variance of the dependent variable that can be explained by the independent variables. An $R^2=1$ is a perfect fit, and an $R^2=0$ means that the coefficients cannot explain any observations.
• The magnitude of a coefficient indicates the relative importance of its independent variable to the model.

Code Listing

Listing (below) provides a code listing of the Ordinary Least Squares method in R. Figure (below) provides a plot of the training dataset with the line of best fit highlighted.

The example uses the lm function in the stats core package which is responsible for fitting linear least squares models [RDCT2011a]. For more information on this library type: library(help="stats"), and for more information on the function type: ?lm.

The test problem is a two-dimensional dataset of 100 samples, where the x-values are drawn from a uniformly random distribution $x \in [0,10]$ and y values are the x value plus a value drawn from a normally random distribution with a mean of 0 and a standard deviation of 1.

# define 2D regression problem of 100 samples
regression_dataset <- function() {
	x <- runif(100, 0, 10)
	y <- x + rnorm(100)
	data.frame(x, y)
}

# get the data
data <- regression_dataset()
# split data in to train and test (67%/33%)
training_set <- sample(100,67)
train <- data[training_set,]
test <- data[(1:100)[-training_set],]

# create a linear regression model using ordinary least squares
model <- lm(
	y~x, # predict Y given X
	train, # training dataset
	NULL, # no weighting on the variables
	NULL, # no action on missing values
	method="qr") # QR decomposition (efficient matrix calculation method)

# summarize the create linear regression model
summary(model)
# regression model diagnostic plots with the training data
par(mfrow=c(2,2))
plot(model)

# plot the training data and the line of best fit
plot(train)
abline(model$coef, lty=5, col="red")

# make predictions for the test data
predictions <- predict.lm(model, test[,1:2])
# compute mean squared error
print(mean((test$y-predictions)^2))

References

Primary Sources

The method for Least Squares may be traced back to Gauss who claims to have devised the method in and later published it in 1809 [Gauss1809] [Gauss1823] (Latin). Legendre independently developed and published the Least Squares method in 1805 [Legendre1805] (Appendix, French).

More Information

Least Squares Regression is covered in any good text of statistics, data analysis or econometrics. Some popular introductory texts to least squares regression include Montgomery et al. [Montgomery2001] (Chapter 2), and Tamhane and Dunlop [Tamhane2000] (Chapter 10). Faraway provides a free book that includes a practical introduction into Least Squares Regression with examples in R [Faraway2002] (updated version published [Faraway2004]).

Bibliography