Loss Functions and Error Metrics

Imagine you are practicing archery. If you shoot and the arrow lands 10 centimeters away from the target center, you measure that distance and adjust your aim on the next attempt. If you never measured how far off you were, you could never improve.

In machine learning, the principle is identical: for a model to learn, we need a precise mathematical metric that scores the discrepancy between the model's predictions and reality. That measure is called the loss functionA mathematical function measuring the difference between predicted values and ground truthView official documentation or cost function.

In this stage, you will implement the two foundational loss functions of supervised learning:

1. Mean Squared Error (MSE)

Widely used in regression tasks and numerical approximation:

$$\text{MSE} = \frac{1}{N} \sum_{i=1}^{N} (p_i - y_i)^2$$

Why do we square the difference?

1. Prevents Cancellation: An error of $+0.5$ on one sample and $-0.5$ on another would sum to zero without squaring, giving the illusion of zero error. Squaring guarantees every deviation is positive.
2. Heavily Penalizes Outliers: A small error of $0.1$ squared becomes $0.01$, but a larger error of $2.0$ squared becomes $4.0$ (a 16x penalty for twice the deviation).

2. Binary Cross-Entropy (BCE)

The canonical loss function for binary classification where targets are boolean labels ($y_i \in \{0, 1\}$) and predictions represent probabilities ($p_i \in (0, 1)$):

$$\text{BCE} = -\frac{1}{N} \sum_{i=1}^{N} \left[ y_i \ln(p_i) + (1 - y_i) \ln(1 - p_i) \right]$$

Why is it effective for classification?

• If the true target is $y=1$, the equation reduces to $-\ln(p)$: predicting $p=0.99$ yields a negligible penalty ($-\ln(0.99) \approx 0.01$). But confidently predicting $p=0.01$ triggers a huge penalty ($-\ln(0.01) \approx 4.60$).
• Numerical Stability ($\epsilon$): Evaluating $\ln(0)$ yields $-\infty$ or NaN. To prevent arithmetic failure, clip $p_i$ to $[\epsilon, 1 - \epsilon]$ with $\epsilon = 10^{-7}$: $$p_i' = \max(10^{-7}, \min(1 - 10^{-7}, p_i))$$ Industry Fact: Production frameworks like PyTorch (torch.nn.BCELoss) perform this exact clipping internally to avoid numerical divergence.

When standard input receives "op": "loss", process:

• metric: string identifier ("mse" or "bce")
• predictions: array of floats $[p_1, \dots, p_N]$
• targets: array of ground truth values $[y_1, \dots, y_N]$

Walkthrough Example (MSE)

{
  "op": "loss",
  "metric": "mse",
  "predictions": [0.8, 0.2],
  "targets": [1.0, 0.0]
}

1. Sample 1 error: $(0.8 - 1.0)^2 = (-0.2)^2 = 0.04$.
2. Sample 2 error: $(0.2 - 0.0)^2 = (0.2)^2 = 0.04$.
3. Mean loss: $\frac{0.04 + 0.04}{2} = 0.04$.
4. Output: {"loss":0.04}.

• MSE Implementation:

  # Python
  mse = sum((p - y) ** 2 for p, y in zip(preds, targets)) / len(preds)
  

  // TypeScript
  const sumSq = preds.reduce((acc, p, i) => acc + (p - targets[i]) ** 2, 0);
  const mse = sumSq / preds.length;
  

• Safe BCE with Numerical Clipping:

  # Python
  eps = 1e-7
  total = 0.0
  for p, y in zip(preds, targets):
      p_c = max(eps, min(1.0 - eps, p))
      total += y * math.log(p_c) + (1.0 - y) * math.log(1.0 - p_c)
  bce = -total / len(preds)
  

  // TypeScript
  const eps = 1e-7;
  let total = 0;
  for (let i = 0; i < preds.length; i++) {
    const p = Math.max(eps, Math.min(1 - eps, preds[i]));
    total += targets[i] * Math.log(p) + (1 - targets[i]) * Math.log(1 - p);
  }
  const bce = -total / preds.length;
  

1. If predictions and targets differ in length or are empty, print error to stderr and exit with code 1.
2. If metric is neither "mse" nor "bce", print error to stderr and exit with code 1.
3. Round the loss to 4 decimal places. If integer (including 0), format as integer.
4. Compact output format:

   { "loss": 0.04 }