[ML Notes] 一元线性回归

1. 基本形式

  一元线性回归的基本形式为:

$$
f(x) = w x + b \tag{1}
$$

其中 $x$ 为特征,$w$ 和 $b$ 分别为权重和偏置。其对应的均方误差为

$$
J(w, b) = \frac{1}{m} \sum_{i=1}^m [y_i – f(x_i)]^2 = \frac{1}{m} \sum_{i=1}^m (y_i – w x_i – b_i)^2 \tag{2}
$$

其中 $m$ 为样例总数,$x_i$ 和 $y_i$ 分别为第 $i$ 个样例的特征和标签。

2. 参数求解

  最佳的 $w$ 和 $b$ 会最小化均方误差,即

$$
(w^*, b^*) = \underset{(w, b)}{\arg\min} \; J(w, b) \tag{3}
$$

  式 $(2)$ 分别为 $w$ 和 $b$ 求偏导,得

$$
\begin{aligned}
\frac{\partial J}{\partial w} &= \sum_{i=1}^m 2(y_i – w x_i + b)(-x) \\
&= 2 \bigg( w \sum_{i=1}^m x_i^2 – \sum_{i=1}^m (y_i – b) x_i \bigg)
\end{aligned} \tag{4}
$$

$$
\begin{aligned}
\frac{\partial J}{\partial b} &= \sum_{i=1}^m 2(y_i – w x_i + b)(-1) \\
&= 2 \bigg( mb – \sum_{i=1}^m (y_i – w x_i) \bigg)
\end{aligned} \tag{5}
$$

由式 $(5)$,令 $\frac{\partial J}{\partial b} = 0$,可解得

$$
b = \frac{1}{m} \sum_{i=1}^m (y_i – w x_i) = \overline{y} – w \overline{x} \tag{6}
$$

由式 $(4)$,令 $\frac{\partial J}{\partial w} = 0$,有

$$
w \sum_{i=1}^m x_i^2 – \sum_{i=1}^m (y_i – b) x_i = 0
$$

将式 $(6)$ 代入上式,得

$$
w \sum_{i=1}^m x_i^2 – \sum_{i=1}^m (y_i – \overline{y} + w \overline{x}) x_i = 0
$$

$$
w \sum_{i=1}^m x_i^2 – \sum_{i=1}^m y_i x_i + \overline{y} \sum_{i=1}^m x_i – w \overline{x} \sum_{i=1}^m x_i = 0
$$

$$
w \bigg( \sum_{i=1}^m x_i^2 – \overline{x} \sum_{i=1}^m x_i \bigg) = \sum_{i=1}^m y_i x_i – \overline{y} \sum_{i=1}^m x_i
$$

$$
w \bigg( \sum_{i=1}^m x_i^2 – \frac{1}{m} \sum_{i=1}^m x_i \sum_{i=1}^m x_i \bigg) = \sum_{i=1}^m y_i x_i – \sum_{i=1}^m y_i \overline{x}
$$

解得

$$
w = \frac{\sum\limits_{i=1}^m y_i (x_i – \overline{x})}{\sum\limits_{i=1}^m x_i^2 – \frac{1}{m} \bigg( \sum\limits_{i=1}^m x_i \bigg)^2 } \tag{7}
$$