Effet de la réponse de commutation et de la variable explicative dans la régression linéaire simple

Disons qu'il existe une "vraie" relation entre et telle que , où et sont des constantes et est un bruit normal. Lorsque je génère de manière aléatoire des données à partir de ce code R: puis que je rentre dans un modèle , je reçois évidemment des estimations raisonnablement bonnes pour et .$y$$x$$y = ax + b + \epsilon$$a$$b$$\epsilon$x <- 1:100; y <- ax + b + rnorm(length(x))y ~ x$a$$b$

(x ~ y)Cependant, si je modifie le rôle des variables comme dans , puis que je réécris le résultat pour que soit fonction de , la pente résultante est toujours plus raide (plus négative ou plus positive) que celle estimée par la régression. J'essaie de comprendre exactement pourquoi et j'apprécierais que quelqu'un me donne une intuition sur ce qui se passe là-bas.$y$$x$y ~ x

Soit points de données , dans le plan, traçons une droite . Si nous prédisons comme la valeur de , alors l' erreur est , l' erreur au carré est , et l' erreur quadratique totale . Nous demandons$n$$(x_i,y_i), i = 1,2,\ldots n$$y = ax+b$$ax_i+b$$\hat{y}_i$$y_i$$(y_i-\hat{y}_i) = (y_i-ax_i-b)$$(y_i-ax_i-b)^2$ $\sum_{i=1}^n (y_i-ax_i-b)^2$

Quel choix de et minimise ?$a$$b$$S =\displaystyle\sum_{i=1}^n (y_i-ax_i-b)^2$

Puisque est la distance verticale de partir de la droite, nous demandons la ligne telle que la somme des carrés des distances verticales des points à partir de la droite soit aussi petite que possible. Maintenant, est une fonction quadratique de et et atteint sa valeur minimale lorsque et sont tels que À partir de la deuxième équation, nous obtenons où $(y_i-ax_i-b)$$(x_i,y_i)$$S$$a$$b$$a$$b$

$$\begin{align*} \frac{\partial S}{\partial a} &= 2\sum_{i=1}^n (y_i-ax_i-b)(-x_i) &= 0\\ \frac{\partial S}{\partial b} &= 2\sum_{i=1}^n (y_i-ax_i-b)(-1) &= 0 \end{align*}$$ $$b = \frac{1}{n}\sum_{i=1}^n (y_i - ax_i) = \mu_y - a\mu_x$$ $\displaystyle \mu_y = \frac{1}{n}\sum_{i=1}^n y_i, ~ \mu_x = \frac{1}{n}\sum_{i=1}^n x_i$ are the arithmetic average values of the $y_i$'s and the $x_i$'s respectively. Substituting into the first equation, we get $$a = \frac{\left(\frac{1}{n}\sum_{i=1}^n x_iy_i\right) -\mu_x\mu_y}{ \left( \frac{1}{n}\sum_{i=1}^n x_i^2\right) -\mu_x^2}.$$ Thus, the line that minimizes $S$ can be expressed as $$y = ax+b = \mu_y + \left(\frac{\left(\frac{1}{n}\sum_{i=1}^n x_iy_i\right) -\mu_x\mu_y}{ \left( \frac{1}{n}\sum_{i=1}^n x_i^2\right) -\mu_x^2}\right) (x - \mu_x),$$ and the minimum value of $S$ is $$S_{\min} = \frac{\left[\left(\frac{1}{n}\sum_{i=1}^n y_i^2\right) -\mu_y^2\right] \left[\left(\frac{1}{n}\sum_{i=1}^n x_i^2\right) -\mu_x^2\right] - \left[\left(\frac{1}{n}\sum_{i=1}^n x_iy_i\right) -\mu_x\mu_y\right]^2}{\left(\frac{1}{n}\sum_{i=1}^n x_i^2\right) -\mu_x^2}.$$

If we interchange the roles of $x$ and $y$, draw a line $x = \hat{a}y + \hat{b}$, and ask for the values of $\hat{a}$ and $\hat{b}$ that minimize

$$T = \sum_{i=1}^n (x_i - \hat{a}y_i - \hat{b})^2,$$ that is, we want the line such that the sum of the squares of the horizontal distances of the points from the line is as small as possible, then we get

$$x = \hat{a}y+\hat{b} = \mu_x + \left(\frac{\left(\frac{1}{n}\sum_{i=1}^n x_iy_i\right) -\mu_x\mu_y}{ \left( \frac{1}{n}\sum_{i=1}^n y_i^2\right) -\mu_y^2}\right) (y - \mu_y)$$ and the minimum value of $T$ is $$T_{\min} = \frac{\left[\left(\frac{1}{n}\sum_{i=1}^n y_i^2\right) -\mu_y^2\right] \left[\left(\frac{1}{n}\sum_{i=1}^n x_i^2\right) -\mu_x^2\right] - \left[\left(\frac{1}{n}\sum_{i=1}^n x_iy_i\right) -\mu_x\mu_y\right]^2}{\left(\frac{1}{n}\sum_{i=1}^n y_i^2\right) -\mu_y^2}.$$

Note that both lines pass through the point $(\mu_x,\mu_y)$ but the slopes are

$$a = \frac{\left(\frac{1}{n}\sum_{i=1}^n x_iy_i\right) -\mu_x\mu_y}{ \left( \frac{1}{n}\sum_{i=1}^n x_i^2\right) -\mu_x^2},~~ \hat{a}^{-1} = \frac{ \left( \frac{1}{n}\sum_{i=1}^n y_i^2\right) -\mu_y^2}{\left(\frac{1}{n}\sum_{i=1}^n x_iy_i\right) -\mu_x\mu_y}$$ are different in general. Indeed, as @whuber points out in a comment, the slopes are the same when all the points $(x_i,y_i)$ lie on the same straight line. To see this, note that $$\hat{a}^{-1} - a = \frac{S_{\min}}{\left(\frac{1}{n}\sum_{i=1}^n x_iy_i\right) -\mu_x\mu_y} = 0 \Rightarrow S_{\min} = 0 \Rightarrow y_i=ax_i+b, i=1,2,\ldots, n.$$

Just to illustrate Dilip’s answer: on the following pictures,

• the black dots are data points ;
• on the left, the black line is the regression line obtained by y ~ x, which minimize the squares of the length of the red segments;
• on the right, the black line is the regression line obtained by x ~ y, which minimize the squares of the length of the red segments.

Edit (least rectangles regression)

If there is no natural way to chose a "response" and a "covariate", but rather the two variables are interdependent you may wish to conserve a symmetrical role for $y$ and $x$; in this case you can use "least rectangles regression."

• write $Y = aX + b + \epsilon$, as usual;
• denote $\hat y_i = a x_i + b$ and $\hat x_i = {1\over a} (y_i - b)$ the estimations of $Y_i$ conditional to $X = x_i$ and of $X_i$ conditional to $Y = y_i$;
• minimize $\sum_i | x_i - \hat x_i | \cdot | y_i - \hat y_i|$, which leads to $$\hat y = \mathrm{sign}\left(\mathrm{cov}(x,y)\right){\hat\sigma_y \over \hat\sigma_x} (x-\overline x) + \overline y.$$

Here is an illustration with the same data points, for each point, a "rectangle" is computed as the product of the length of two red segments, and the sum of rectangles is minimized. I don’t know much about the properties of this regression and I don’t find much with google.

Just a brief note on why you see the slope smaller for one regression. Both slopes depend on three numbers: standard deviations of $x$ and $y$ ($s_{x}$ and $s_{y}$), and correlation between $x$ and $y$ ($r$). The regression with $y$ as response has slope $r\frac{s_{y}}{s_{x}}$ and the regression with $x$ as response has slope $r\frac{s_{x}}{s_{y}}$, hence the ratio of the first slope to the reciprocal of the second is equal to $r^2\leq 1$.

So the greater the proportion of variance explained, the closer the slopes obtained from each case. Note that the proportion of variance explained is symmetric and equal to the squared correlation in simple linear regression.

A simple way to look at this is to note that, if for the true model $y=\alpha+\beta x+\epsilon$, you run two regressions:

• $y=a_{y\sim x}+b_{y\sim x} x$
• $x=a_{x\sim y}+b_{x\sim y} y$

Then we have, using $b_{y\sim x}=\frac{cov(x,y)}{var(x)}=\frac{cov(x,y)}{var(y)}\frac{var(y)}{var(x)}$:

$$b_{y\sim x}=b_{x\sim y}\frac{var(y)}{var(x)}$$

So whether you get a steeper slope or not just depends on the ratio $\frac{var(y)}{var(x)}$. This ratio is equal to, based on the assumed true model:

$$\frac{var(y)}{var(x)}=\frac{\beta^2 var(x) + var(\epsilon)}{var(x)}$$

Link with other answers

You can connect this result with the answers from others, who said that when $R^2=1$, it should be the reciprocal. Indeed, $R^2=1\Rightarrow var(\epsilon) = 0$, and also, $b_{y\sim x}=\beta$ (no estimation error), Hence:

$$R^2=1\Rightarrow b_{y\sim x}=b_{x\sim y}\frac{\beta^2 var(x) + 0}{var(x)}=b_{x\sim y}\beta^2$$

So $b_{x\sim y}=1/\beta$

It becomes interesting when there is also noise on your inputs (which we could argue is always the case, no command or observation is ever perfect).

I have built some simulations to observe the phenomenon, based on a simple linear relationship $x = y$, with Gaussian noise on both x and y. I generated the observations as follows (python code):

x = np.linspace(0, 1, n)
y = x

x_o = x + np.random.normal(0, 0.2, n)
y_o = y + np.random.normal(0, 0.2, n)

See the different results (odr here is orthogonal distance regression, i.e. the same as least rectangles regression):

All the code is in there:

https://gist.github.com/jclevesque/5273ad9077d9ea93994f6d96c20b0ddd

Regression line is not (always) the same as true relationship

You may have some 'true' causal relationship like

$$y = a + bx + \epsilon$$

but fitted regression lines y ~ x or x ~ y do not mean the same as that causal relationship (even when in practice the expression for one of the regression line may coincide with the expression for the causal 'true' relationship)

---

More precise relationship between slopes

For two switched simple linear regressions:

$$Y = a_1 + b_1 X\\X = a_2 + b_2 Y$$

you can relate the slopes as following:

$$b_1 = \rho^2 \frac{1}{b_2} \leq \frac{1}{b_2}$$

So the slopes are not each other inverse.

---

Intuition

The reason is that

• Regression lines and correlations do not necessarily correspond one-to-one to a causal relationship.
• Regression lines relate more directly to a conditional probability or best prediction.

You can imagine that the conditional probability relates to the strength of the relationship. Regression lines reflect this and the slopes of the lines may be both shallow when the strength of the relationship is small or both steep when the strength of the relationship is strong. The slopes are not simply each others inverse.

Example

If two variables $X$ and $Y$ relate to each other by some (causal) linear relationship

$$Y = \text{a little bit of $X + $ a lot of error}$$ Then you can imagine that it would not be good to entirely reverse that relationship in case you wish to express $X$ based on a given value of $Y$.

Instead of

$$X = \text{a lot of $Y + $ a little of error}$$

it would be better to also use

$$X = \text{a little bit of $Y + $ a lot of error}$$

See the following example distributions with their respective regression lines. The distributions are multivariate normal with $\Sigma_{11} \Sigma_{22}=1$ and $\Sigma_{12} = \Sigma_{21} = \rho$

The conditional expected values (what you would get in a linear regression) are

$$\begin{array}{} E(Y|X) &=& \rho X \\ E(X|Y) &=& \rho Y \end{array}$$

and in this case with $X,Y$ a multivariate normal distribution, then the marginal distributions are

$$\begin{array}{} Y & \sim & N(\rho X,1-\rho^2) \\ X & \sim & N(\rho Y,1-\rho^2) \end{array}$$

So you can see the variable Y as being a part $\rho X$ and a part noise with variance $1-\rho^2$. The same is true the other way around.

The larger the correlation coefficient $\rho$, the closer the two lines will be. But the lower the correlation, the less strong the relationship, the less steep the lines will be (this is true for both lines Y ~ X and X ~ Y)

The short answer

The goal of a simple linear regression is to come up with the best predictions of the y variable, given values of the x variable. This is a different goal than trying to come up with the best prediction of the x variable, given values of the y variable.

Simple linear regression of y ~ x gives you the 'best' possible model for predicting y given x. Hence, if you fit a model for x ~ y and algebraically inverted it, that model could at its very best do only as well as the model for y ~ x. But inverting a model fit for x ~ y will usually do worse at predicting y given x, compared to the 'optimal' y ~ x model, because the "inverted x ~ y model" was created to fulfill a different objective.

Illustration

Imagine you have the following dataset:

When you run an OLS regression of y ~ x, you come up with the following model

y = 0.167 + 1.5*x

This optimizes predictions of y by making the following predictions, which have associated errors:

The OLS regression's predictions are optimal in the sense that the sum of the values in the rightmost column (i.e. the sum of squares) is as small as can be.

When you run an OLS regression of x ~ y, you come up with a different model:

x = -0.07 + 0.64*y

This optimizes predictions of x by making the following predictions, with associated errors.

Again, this is optimal in the sense that the sum of the values of the rightmost column are as small as possible (equal to 0.071).

Now, imagine you tried to just invert the first model, y = 0.167 + 1.5*x, using algebra, giving you the model x = -0.11 + 0.67*x.

This would give you the following predictions and associated errors:

The sum of the values in the rightmost column is 0.074, which is larger than the corresponding sum from the model you get from regressing x on y, i.e. the x ~ y model. In other words, the "inverted y ~ x model" is doing a worse job at predicting x than the OLS model of x ~ y.