#machine-learning, eth/cil/theory

Evidence Lower Bound

yields approximate, often tractable objective functions

Definition

l (θ) = \sum_{t = 1}^{s} \ln \sum_{z = 1}^{k} π_{z} p (x; θ_{z}) \geq \sum_{t = 1}^{s} \sum_{z = 1}^{k} q_{t z} [\ln π_{z} + \ln p (x_{t}; θ_{z}) - \ln q_{t z}]

Where $q$ is a vector of convex weights for each sample, ( $q_{t k}$ denotes the probability that sample $t$ is part of the $k$ -th cluster, intuitively)

Derive ELBO (CIL-style)

Starting with some loss function $l (θ)$ :

\begin{aligned} l (θ) = \sum_{t = 1}^{s} \ln \sum_{z = 1}^{k} π_{z} p (x; θ_{z}) & = \sum_{t = 1}^{s} \ln \sum_{z = 1}^{k} \frac{q_{t z} π_{z} p (x; θ_{z})}{q_{t z}} \\ \geq \sum_{t = 1}^{s} \sum_{z = 1}^{k} q_{t z} \ln \frac{π_{z} p (x; θ_{z})}{q_{t z}} \\ = \sum_{t = 1}^{s} \sum_{z = 1}^{k} q_{t z} (\ln π_{z} + \ln p (x; θ_{z}) - \ln q_{t z}) \end{aligned}

Where we used Jensens inequality with $q_{t z}$ as the positive parameters:

jensen-inequality

Jensen Inequality

Definition Finite Form

For real convex function $φ$ and positive weights $a_{i}$
$φ (\frac{\sum a_{i} x_{i}}{\sum a_{i}}) \leq (\frac{\sum a_{i} φ (x_{i})}{\sum a_{i}})$