KL Divergence

KL Divergence ( Kullback-Leibler Divergence)

KL Divergence, also known as Relative Entropy, measures the amount of difference between two probability distributions. This metric can give you insights about how similar or dissimilar the two distributions are.

Cross Entropy: Binary Case

In a binary situation, there are only two possible outcomes: 0 or 1. The measure used to determine the "distance" or difference between the true and predicted values in this case is called Binary Cross Entropy (BCE).

Equation

$BCE = -y\log(\hat{y}) - (1 - y)\log(1 - \hat{y})$

Where:

• $y$ = true target value

• $\hat{y}$ = predicted model value

case 1.

$y=0, \hat{y}=0$

target value matches with the model

$BCE=−ylog(\hat{y})−(1−y)log(1−\hat{y}) \\=1⋅log(1)−0⋅log(0)=0$

case 2

$y=0, \hat{y}=1$

target value doesn't match with the model

$BCE=−ylog(\hat{y})−(1−y)log(1−\hat{y}) \\=−1⋅log(0)−0⋅log(1)=∞$

case3

$y=1, \hat{y}=0$

target value doesn't match with the model

$BCE=−ylog(\hat{y})−(1−y)log(1−\hat{y}) \\=−0⋅log(1)−(1)⋅log(0)=∞$

case4

$y=1, \hat{y}=1$

target value matches with the model

$BCE=−ylog(\hat{y})−(1−y)log(1−\hat{y}) \\=−0⋅log(0)−(1)log(1)=0$

Cross Entropy: Multiple Cases

When we're dealing with more than two possible outcomes, the Binary Cross Entropy formula needs to be adjusted.

Equation:

$CE = - \displaystyle\sum_{x \in \chi} P(x) \log(Q(x))$

Where:

• $P(x)$ = true probability distribution

• $Q(x)$ = predicted probability distribution

Abnormal dice cross entropy

$⇒−\displaystyle∑_iP(x_i)log(Q(x_i))=−(\frac{1}{6}log_2(\frac{1}{2})+\frac{5}{6}log_2(\frac{1}{10}))=2.9349 \ bit(8)$

Normal dice cross entropy

$⇒−\displaystyle∑_iP(x_i)log(Q(x_i))=−(log_2(\frac{1}{6}))= 2.5850 \ bit(8)$

KL Divergence: Relative Comparison of Information Entropy

KL Divergence, also called Relative Entropy, shows the difference between two distributions. The key concept here is "divergence", which denotes the level of difference between the two distributions.

Equation:

$DKL(P||Q) = \displaystyle\sum_{x \in \chi} P(x) \log\left(\frac{P(x)}{Q(x)}\right)$

This can be expanded and rewritten as:

$DKL(P||Q) = - \displaystyle\sum_{x \in \chi} P(x) \log\left(\frac{Q(x)}{P(x)}\right)$

And further simplified into:

$−\displaystyle∑_{x∈χ}P(x)log_b⁡(\frac{Q(x)}{P(x)})=−\displaystyle∑_{x∈χ}P(x)log_bQ(x)+∑_{x∈χ}P(x)log_bP(x)$

$⇒−E_P[log_bQ(x)]+E_P[log_bP(x)]$

$⇒H_p(Q)-H_P(P)$

Conclusion

These formulas and concepts are the backbone of many machine learning algorithms and are essential for understanding how models learn from data and make predictions.