The Training Epoch and Backpropagation

BackpropagationAn algorithm for calculating the gradient of the loss function with respect to weights using the chain ruleView official documentation is often portrayed as esoteric, but at its heart lies a straightforward intuition: tracing the root cause of an error backwards.

Imagine a gear train: you turn an input dial (weight $w$), which rotates an internal gear (linear combination $z$), which opens a pressure valve (sigmoid activation $\hat{y}$), which affects the final reading on a gauge (loss $L$). If the final gauge reading is off, how much should you turn the input dial?

The chain rule of calculus states that we simply multiply the rates of change along each link in the pipeline:

$$\frac{\partial L}{\partial w_j} = \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial z} \cdot \frac{\partial z}{\partial w_j}$$

Breaking Down Each Link Step by Step:

1. Link 1 (How loss changes with respect to prediction): For an individual squared loss $L = \frac{1}{2}(\hat{y} - y)^2$, its derivative is: $$\frac{\partial L}{\partial \hat{y}} = \hat{y} - y$$
2. Link 2 (How sigmoid activation changes with respect to $z$): One of the most elegant mathematical properties of the sigmoid function is that its derivative can be expressed directly in terms of its output: $$\frac{\partial \hat{y}}{\partial z} = \hat{y} \cdot (1 - \hat{y})$$
3. Link 3 (How the affine combination changes with respect to weight $w_j$ or bias $b$): Since $z = w_1 x_1 + \dots + w_j x_j + b$: $$\frac{\partial z}{\partial w_j} = x_j, \quad \frac{\partial z}{\partial b} = 1$$

Multiplying the first two links yields the local sample error term $\delta$:

$$\delta = (\hat{y} - y) \cdot \hat{y} \cdot (1 - \hat{y})$$

Consequently, the gradients for each parameter on a single sample are:

$$\text{Gradient of } w_j = \delta \cdot x_j$$ $$\text{Gradient of } b = \delta \cdot 1 = \delta$$

An epoch is one complete pass through the entire training dataset:

1. For every sample in the batch, compute the forward pass, evaluate loss, and accumulate gradients.
2. After iterating through all samples, compute the mean gradient by dividing by total samples $N$.
3. Update weights and bias using the gradient descent rule implemented in the previous stage.

When standard input contains "op": "train-epoch", process:

• learning_rate: learning rate $\alpha$
• weights: current weights list $[w_1, \dots, w_d]$
• bias: current bias $b$
• samples: list of training samples formatted as {"x": [x_1, ..., x_d], "y": y_target}

Step-by-Step Algorithm

1. Initialize total_loss = 0.0, grad_w = [0.0] * d, grad_b = 0.0.
2. For each sample s in samples:
   • Compute $z = \sum_{j=1}^{d} (w_j \cdot s.x_j) + b$.
   • Compute $\hat{y} = \frac{1}{1 + e^{-z}}$.
   • Add sample loss to total_loss: $0.5 \times (\hat{y} - s.y)^2$.
   • Compute delta: $\delta = (\hat{y} - s.y) \times \hat{y} \times (1.0 - \hat{y})$.
   • For each feature $j$: accumulate $\delta \times s.x_j$ into grad_w[j].
   • Accumulate $\delta$ into grad_b.
3. Compute batch averages dividing by $N = \text{len}(samples)$:
   • mean_loss = total_loss / N
   • grad_w_mean[j] = grad_w[j] / N
   • grad_b_mean = grad_b / N
4. Update parameters:
   • $w_j' = w_j - \alpha \cdot \text{grad\_w\_mean}[j]$
   • $b' = b - \alpha \cdot \text{grad\_b\_mean}$

• Python Pattern:

  n = len(samples)
  d = len(weights)
  total_loss = 0.0
  grad_w = [0.0] * d
  grad_b = 0.0
  
  for s in samples:
      x, y = s["x"], s["y"]
      z = sum(xi * wi for xi, wi in zip(x, weights)) + bias
      y_hat = 1.0 / (1.0 + math.exp(-z))
      total_loss += 0.5 * ((y_hat - y) ** 2)
      delta = (y_hat - y) * y_hat * (1.0 - y_hat)
      for j in range(d):
          grad_w[j] += delta * x[j]
      grad_b += delta
  
  new_w = [round_value(weights[j] - lr * (grad_w[j] / n)) for j in range(d)]
  new_b = round_value(bias - lr * (grad_b / n))
  mean_loss = round_value(total_loss / n)
  
  return {
      "bias": new_b,
      "loss": mean_loss,
      "weights": new_w
  }
  

• TypeScript Pattern:

  const n = samples.length;
  const d = weights.length;
  let totalLoss = 0;
  const gradW = new Array(d).fill(0);
  let gradB = 0;
  
  for (const s of samples) {
    let z = bias;
    for (let j = 0; j < d; j++) z += s.x[j] * weights[j];
    const yHat = 1 / (1 + Math.exp(-z));
    totalLoss += 0.5 * (yHat - s.y) ** 2;
    const delta = (yHat - s.y) * yHat * (1 - yHat);
    for (let j = 0; j < d; j++) gradW[j] += delta * s.x[j];
    gradB += delta;
  }
  
  const newW = weights.map((w, j) => roundValue(w - lr * (gradW[j] / n)));
  const newB = roundValue(bias - lr * (gradB / n));
  const meanLoss = roundValue(totalLoss / n);
  
  return {
    bias: newB,
    loss: meanLoss,
    weights: newW,
  };
  

1. If samples is empty or any sample vector dimension differs from weights, write error to stderr and exit with code 1.
2. Round all floats to 4 decimal places with round_value / roundValue (serialize integers or zero without decimal points, e.g. 0).
3. Output compact JSON sorted alphabetically by key ("bias", "loss", "weights"):

   { "bias": 0, "loss": 0.125, "weights": [0.0063, -0.0063] }