Processing math: 13%

[ML Notes] 一元线性回归

1. 基本形式

  一元线性回归的基本形式为:

f(x)=wx+b

其中 x 为特征,wb 分别为权重和偏置。其对应的均方误差为

J(w, b) = \frac{1}{m} \sum_{i=1}^m [y_i – f(x_i)]^2 = \frac{1}{m} \sum_{i=1}^m (y_i – w x_i – b_i)^2 \tag{2}

其中 m 为样例总数,x_iy_i 分别为第 i 个样例的特征和标签。

2. 参数求解

  最佳的 wb 会最小化均方误差,即

(w^*, b^*) = \underset{(w, b)}{\arg\min} \; J(w, b) \tag{3}

  式 (2) 分别为 wb 求偏导,得

\begin{aligned} \frac{\partial J}{\partial w} &= \sum_{i=1}^m 2(y_i – w x_i + b)(-x) \\ &= 2 \bigg( w \sum_{i=1}^m x_i^2 – \sum_{i=1}^m (y_i – b) x_i \bigg) \end{aligned} \tag{4}

\begin{aligned} \frac{\partial J}{\partial b} &= \sum_{i=1}^m 2(y_i – w x_i + b)(-1) \\ &= 2 \bigg( mb – \sum_{i=1}^m (y_i – w x_i) \bigg) \end{aligned} \tag{5}

由式 (5),令 \frac{\partial J}{\partial b} = 0,可解得

b = \frac{1}{m} \sum_{i=1}^m (y_i – w x_i) = \overline{y} – w \overline{x} \tag{6}

由式 (4),令 \frac{\partial J}{\partial w} = 0,有

w \sum_{i=1}^m x_i^2 – \sum_{i=1}^m (y_i – b) x_i = 0

将式 (6) 代入上式,得

w \sum_{i=1}^m x_i^2 – \sum_{i=1}^m (y_i – \overline{y} + w \overline{x}) x_i = 0

w \sum_{i=1}^m x_i^2 – \sum_{i=1}^m y_i x_i + \overline{y} \sum_{i=1}^m x_i – w \overline{x} \sum_{i=1}^m x_i = 0

w \bigg( \sum_{i=1}^m x_i^2 – \overline{x} \sum_{i=1}^m x_i \bigg) = \sum_{i=1}^m y_i x_i – \overline{y} \sum_{i=1}^m x_i

w \bigg( \sum_{i=1}^m x_i^2 – \frac{1}{m} \sum_{i=1}^m x_i \sum_{i=1}^m x_i \bigg) = \sum_{i=1}^m y_i x_i – \sum_{i=1}^m y_i \overline{x}

解得

w = \frac{\sum\limits_{i=1}^m y_i (x_i – \overline{x})}{\sum\limits_{i=1}^m x_i^2 – \frac{1}{m} \bigg( \sum\limits_{i=1}^m x_i \bigg)^2 } \tag{7}