[ML Notes] 一元线性回归
Contents [show]
1. 基本形式
一元线性回归的基本形式为:
f(x)=wx+b
其中 x 为特征,w 和 b 分别为权重和偏置。其对应的均方误差为
J(w, b) = \frac{1}{m} \sum_{i=1}^m [y_i – f(x_i)]^2 = \frac{1}{m} \sum_{i=1}^m (y_i – w x_i – b_i)^2 \tag{2}
其中 m 为样例总数,x_i 和 y_i 分别为第 i 个样例的特征和标签。
2. 参数求解
最佳的 w 和 b 会最小化均方误差,即
(w^*, b^*) = \underset{(w, b)}{\arg\min} \; J(w, b) \tag{3}
式 (2) 分别为 w 和 b 求偏导,得
\begin{aligned} \frac{\partial J}{\partial w} &= \sum_{i=1}^m 2(y_i – w x_i + b)(-x) \\ &= 2 \bigg( w \sum_{i=1}^m x_i^2 – \sum_{i=1}^m (y_i – b) x_i \bigg) \end{aligned} \tag{4}
\begin{aligned} \frac{\partial J}{\partial b} &= \sum_{i=1}^m 2(y_i – w x_i + b)(-1) \\ &= 2 \bigg( mb – \sum_{i=1}^m (y_i – w x_i) \bigg) \end{aligned} \tag{5}
由式 (5),令 \frac{\partial J}{\partial b} = 0,可解得
b = \frac{1}{m} \sum_{i=1}^m (y_i – w x_i) = \overline{y} – w \overline{x} \tag{6}
由式 (4),令 \frac{\partial J}{\partial w} = 0,有
w \sum_{i=1}^m x_i^2 – \sum_{i=1}^m (y_i – b) x_i = 0
将式 (6) 代入上式,得
w \sum_{i=1}^m x_i^2 – \sum_{i=1}^m (y_i – \overline{y} + w \overline{x}) x_i = 0
w \sum_{i=1}^m x_i^2 – \sum_{i=1}^m y_i x_i + \overline{y} \sum_{i=1}^m x_i – w \overline{x} \sum_{i=1}^m x_i = 0
w \bigg( \sum_{i=1}^m x_i^2 – \overline{x} \sum_{i=1}^m x_i \bigg) = \sum_{i=1}^m y_i x_i – \overline{y} \sum_{i=1}^m x_i
w \bigg( \sum_{i=1}^m x_i^2 – \frac{1}{m} \sum_{i=1}^m x_i \sum_{i=1}^m x_i \bigg) = \sum_{i=1}^m y_i x_i – \sum_{i=1}^m y_i \overline{x}
解得
w = \frac{\sum\limits_{i=1}^m y_i (x_i – \overline{x})}{\sum\limits_{i=1}^m x_i^2 – \frac{1}{m} \bigg( \sum\limits_{i=1}^m x_i \bigg)^2 } \tag{7}