Cross Entropy Loss

Cross-Entropy measures the distance between two probability distributions: the true labels and the predicted probabilities.

Formula (for one example with C classes):

$$\mathcal{L}=-\sum _{i=1}^C{y}_i\log ({\hat{y}}_i)$$

Where:

• $y_i$ = ground truth (one-hot encoded, 0 or 1)
• ${\hat{y}}_i$ = predicted probability for class $i$.

Simplified (when $y$ is one-hot):

$$\mathcal{L}=-\log ({\hat{y}}_{\text{true class}})$$

For a batch of N examples:

$${\mathcal{L}}_{\text{batch}}=\frac{1}{N}\sum _{n=1}^N{\mathcal{L}}^{(n)}$$

---

Intuition:

• High loss if the predicted probability for the correct class is low.
• Low loss if the predicted probability for the correct class is high.