Introduction to Multiple and Logistic Regression

The principles of simple linear regression lay the foundation for more sophisticated regression methods used in a wide range of challenging settings. In this section, we explore multiple regression, which introduces the possibility of more than one predictor, and logistic regression, a technique for predicting categorical outcomes with two possible categories.

Multiple regression extends simple two-variable regression to the case that still has one response but many predictors (denoted x₁, x₂, x₃, …). The method is motivated by scenarios where many variables may be simultaneously connected to an output.

We will consider eBay auctions of a video game called Mario Kart for the Nintendo Wii. The outcome variable of interest is the total price of an auction, which is the highest bid plus the shipping cost. We will try to determine how total price is related to each characteristic in an auction while simultaneously controlling for other variables. For instance, all other characteristics held constant, are longer auctions associated with higher or lower prices? And, on average, how much more do buyers tend to pay for additional Wii wheels (plastic steering wheels that attach to the Wii controller) in auctions? Multiple regression will help us answer these and other questions.

The data set mario_kart includes results from 141 auctions.^[1] Four observations from this data set are shown in Table 1, and descriptions for each variable are shown in Table 2. Notice that the condition and stock photo variables are indicator variables. For instance, the cond_new variable takes value 1 if the game up for auction is new and 0 if it is used. Using indicator variables in place of category names allows for these variables to be directly used in regression. Multiple regression also allows for categorical variables with many levels, though we do not have any such variables in this analysis, and we save these details for a second or third course.

A single-variable model for the Mario Kart data

Let’s fit a linear regression model with the game’s condition as a predictor of auction price. The model may be written as

[latex]\widehat{\text{price}}=42.87 + 10.90\times\text{cond_new}[/latex]

Results of this model are shown in Table 3 and a scatterplot for price versus game condition.

Including and assessing many variables in a model

Sometimes there are underlying structures or relationships between predictor variables. For instance, new games sold on Ebay tend to come with more Wii wheels, which may have led to higher prices for those auctions. We would like to fit a model that includes all potentially important variables simultaneously. This would help us evaluate the relationship between a predictor variable and the outcome while controlling for the potential influence of other variables. This is the strategy used in multiple regression. While we remain cautious about making any causal interpretations using multiple regression, such models are a common first step in providing evidence of a causal connection.

We want to construct a model that accounts for not only the game condition, as in the mario_kart example, but simultaneously accounts for three other variables: stock photo, duration, and wheels.

[latex]\begin{array}\widehat{\text{price}}\hfill &={\beta}_{0}\hfill &+{\beta}_{1}\times\text{cond_new}\hfill&+{\beta}_{2}\times\text{stock_photo}\text{ }\hfill &+{\beta}_{3}\times\text{duration}\hfill&+{\beta}_{4}\hfill&\times\text{wheels}\\\hat{y}\hfill &={\beta}_{0}\hfill &+{\beta}_{1}{x}_{1}\hfill &+{\beta}_{2}{x}_{2}\hfill &+{\beta}_{3}{x}_{3}\hfill &+{\beta}_{4}{x}_{4}\end{array}[/latex]

In this equation, y represents the total price, x₁  indicates whether the game is new, x₂ indicates whether a stock photo was used, x₃ is the duration of the auction, and x₄ is the number of Wii wheels included with the game. Just as with the single predictor case, a multiple regression model may be missing important components or it might not precisely represent the relationship between the outcome and the available explanatory variables. While no model is perfect, we wish to explore the possibility that this one may fit the data reasonably well.

We estimate the parameters [latex]{\beta}_{0},{\beta}_{1},\dots,{\beta}_{4}[/latex]  in the same way as we did in the case of a single predictor. We select [latex]{b}_{0},{b}_{1},\dots,{b}_{4}[/latex]  that minimize the sum of the squared residuals:

SSE = [latex]\displaystyle{{e}_{1}}^{2}+{{e}_{2}}^{2}+\dots+{{e}_{141}}^{2}={\sum}_{i = 1}^{141}{\left({{e}_{i}}\right)}^{2} = {\sum}_{i = 1}^{141}{\left({{y}_{i} - {\hat{y}}_{i}}\right)}^{2}[/latex]

Here there are 141 residuals, one for each observation. We typically use a computer to minimize the SSE and compute point estimates, as shown in the sample output in the table below. Using this output, we identify the point estimates b_i of each i, just as we did in the one-predictor case.

Example 2 describes a common issue in multiple regression: correlation among predictor variables. We say the two predictor variables are collinear (pronounced as co-linear ) when they are correlated, and this collinearity complicates model estimation. While it is impossible to prevent collinearity from arising in observational data, experiments are usually designed to prevent predictors from being collinear.

Adjusted R² as a better estimate of explained variance

We first used R² to determine the amount of variability in the response that was explained by the model:

[latex]\displaystyle{R}^2=1-\frac{\text{variability in residuals}}{\text{variability in the outcome}}=1=\frac{\text{Var}(e_i)}{\text{Var}(y_i)}[/latex]

where e_i represents the residuals of the model and y_i the outcomes. This equation remains valid in the multiple regression framework, but a small enhancement can often be even more informative.

This strategy for estimating R² is acceptable when there is just a single variable. However, it becomes less helpful when there are many variables. The regular R² is a less estimate of the amount of variability explained by the model. To get a better estimate, we use the adjusted R².

Because k is never negative, the adjusted R² will be smaller—often times just a little smaller—than the unadjusted R². The reasoning behind the adjusted R² lies in the degrees of freedom associated with each variance.^[2]

Adjusted R² could have been used earlier. However, when there is only k = 1 predictors, adjusted R² is very close to regular R², so this nuance isn’t typically important when considering only one predictor.