AMATH 515 Lecture 3 - GLM Family

Lecture 3 -

Examples from GLM family:

\begin{equation} \min_x G(Ax) - b^T Ax \end{equation}

\begin{equation} G(z) = g(z_1) + \cdots + g(z_n) \end{equation}

Gaussian

\begin{equation} g(z) = \frac{1}{2} z^2 \end{equation}

Logistic

\begin{equation} g(z) = \log(1 + \exp(z)) \end{equation}

Poisson

\begin{equation} g(z) = \exp(z) \end{equation}

Given differentiable function $f:\mathbb{R}^n \rightarrow \mathbb{R}$, we want to solve $\min f(x)$.

Gaussian case:

\begin{equation} \min \frac{1}{2} ||Ax||^2 - b^T Ax \end{equation}

\begin{equation} A^T Ax - A^T b = 0 \end{equation}

Solve for x gives

\begin{equation} x = (A^T A)^{-1} A^T b \end{equation}

Linear model for $f$, using $\nabla f$

\begin{equation} f(x) = f(x_0) + \nabla f(x_0)^T (x - x_0) + o(||x - x_0||) \end{equation}

Finding $\alpha$

In practice, look for $\alpha$ that ensures

\begin{equation} f(x + \alpha d) \leq f(x_0) - \alpha(0.01) || \nabla f(x_0) || \end{equation}