ML:逻辑回归的梯度下降算法

ML：极大似然估计

概率密度（质量）函数：用来描述随机变量取某个值的时候，取值点对应的概率的函数。

概率：已知概率分布，推断样本的概率值

似然：已经有观测样本，寻找最符合当前数据分布的参数

似然函数：$\mathcal{L}(\mu, \sigma | X)=\prod_{i=1}^{N} P\left(x_{i} | \mu, \sigma\right)$

对数似然函数：$\mathcal{L}(\mu, \sigma | X)=\sum_{i=1}^{N} \log P\left(x_{i} | \mu, \sigma\right)$

损失函数：$J(\theta)=-\sum_{i}^{m} Y \log (\hat{Y})-(1-Y) \log (1-\hat{Y})$，需要求$J(\theta)$对于$\theta_{i}$的导数，式中$\hat{Y}=\frac{1}{1+e^{-\theta^{T} X}}$

利用$\frac{d}{d x} \log _{a}(f(x))=\frac{1}{f(x) \ln a} f^{\prime}(x)$，将$\hat{Y}=\frac{1}{1+e^{-\theta^{T} X}}$代入$log(\hat{Y})$:

$\frac{\partial}{\partial \theta_{j}} \log (\hat{Y})=\frac{\partial}{\partial \theta_{j}} \log \left(\frac{1}{1+e^{-\theta^{T} x}}\right)=\frac{\partial}{\partial \theta_{j}}(\log (1)-\log \left(1+e^{-\theta^{T} x}\right))$

$\frac{\partial}{\partial \theta_{j}} \log (\hat{Y})=\frac{\partial}{\partial \theta_{j}}(-\log \left(1+e^{-\theta^{T} x}\right))=-\frac{1}{1+e^{-\theta^{T} x}} \cdot e^{-\theta^{T} x} \cdot-x_{j}=\left(1-\frac{1}{1+e^{-\theta^{T} x}}\right) x$

$\frac{\partial}{\partial \theta_{j}} \log (1-\hat{Y})=\frac{\partial}{\partial \theta_{j}} \log \left(\frac{e^{-\theta^{T} x}}{1+e^{-\theta^{T} x}}\right)=\frac{\partial}{\partial \theta_{j}}(-\theta^{T} x-\log \left(1+e^{-\theta^{T} x}\right))$

$\frac{\partial}{\partial \theta_{j}} \log (1-\hat{Y})=-x_{j}+x_{j}\left(1-\frac{1}{1+e^{-\theta^{T} x}}\right)=-\frac{1}{1+e^{-\theta^{T} x}} x_{j}$

综上可得，$\frac{\partial}{\partial \theta_{j}} J(\theta)=-\sum_{i}^{m} y_{i} x_{i j}\left(1-\frac{1}{1+e^{-\theta^{T} x_{i}}}\right)-\left(1-y_{i}\right) x_{i j} \frac{1}{1+e^{-\theta^{T} x_{i}}}$

其中，$i$是数据点的序号，$j$是特征的数量，输入$X$可以表示为：

$X=\left[\begin{array}{ll}{x_{i=1, j=1}} & {x_{i=2, j=1} x_{i=3, j=1}} \\ {x_{i=1, j=2}} & {x_{i=2, j=2} x_{i=3, j=2}} \\ {x_{i=1, j=3}} & {x_{i=2, j=3} x_{i=3, j=3}}\end{array}\right]$，举个例子，一个batch的图片，$x_{i j}$表示第$i$张图片的第$j$个像素

展开整理得：$\frac{\partial}{\partial \theta_{j}} J(\theta)=\sum_{i}^{m}\left(\frac{1}{1+e^{-\theta^{T} x_{i}}}-y_{i}\right) x_{i j}=\sum_{i}^{m}\left(\hat{y}_{i}-y_{i}\right) x_{i j}$，式中$\hat{Y}=\frac{1}{1+e^{-\theta^{T} X}}$

之前$\theta$和$X$的向量表示形式为：$\theta^{T}=\left[\begin{array}{lll}{\text { bias }} & {\theta_{1}} & {\theta_{2}}\end{array}\right]$和$X=\left[\begin{array}{c}{1} \\ {x_{1}} \\ {x_{2}}\end{array}\right]$

由于$\theta$中的$bias$对应着$X$里面的1，所以可以得到：$\frac{\partial}{\partial b i a s} J(\theta)=\sum_{i}^{m}\left(\hat{y}_{i}-y_{i}\right)$

设定学习率$\eta$，迭代下面的步骤直至收敛：

$\theta_{j} \leftarrow \theta_{j}-\eta \frac{\partial}{\partial \theta_{j}} J(\theta)$

$bias \leftarrow bias -\eta \frac{\partial}{\partial \text { bias }} J(\theta)$

来源：CSDN

作者：三少Algorithm

链接：https://blog.csdn.net/weixin_42717395/article/details/103246496