Gradient Descent Parameter Update

How do a collection of raw numbers actually "learn"? It all comes down to an intuitive geometric insight: gradient descentAn optimization algorithm for finding a local minimum of a differentiable functionView official documentation.

Imagine you are standing on a steep mountainside surrounded by thick fog, with zero visibility beyond your feet. Your mission is to reach the valley floor where elevation (the loss or error) is lowest:

1. You cannot see the global valley from where you stand, but you can feel the slope of the ground right under your boots.
2. That slope is the derivative or gradient ($\nabla$): it points in the direction of steepest ascent (uphill).
3. To walk downhill towards the minimum error, you must take a step in the exact opposite direction (with a negative sign: $-\nabla$).
4. The size of the step you take is determined by the learning rate ($\alpha$).

High loss position (error)
     \
      \  <-- Steps taken opposite to gradient (-α·∇)
       \
        \____  Valley floor: Minimum loss (optimal learning)

The Learning Rate ($\alpha$) Dilemma

• Too small ($\alpha = 0.0001$): Steps are overly conservative; the network requires millions of iterations to learn simple patterns.
• Too large ($\alpha = 10.0$): You risk leaping entirely over the valley floor, landing on an even steeper opposing cliff (causing divergence).
• Balanced ($\alpha = 0.1$): The parameters steadily converge toward the minimum loss.

Every weight and bias is updated by subtracting the product of the learning rate and its respective gradient:

$$w_i' = w_i - \alpha \cdot \nabla w_i$$ $$b' = b - \alpha \cdot \nabla b$$

When standard input receives "op": "gradient-step", process:

{
  "op": "gradient-step",
  "learning_rate": 0.1,
  "weights": [0.5, -0.2],
  "bias": 0.1,
  "grad_weights": [0.2, -0.4],
  "grad_bias": 0.05
}

Step-by-Step Walkthrough

1. Update the first weight: $$w_1' = 0.5 - (0.1 \times 0.2) = 0.5 - 0.02 = 0.48$$
2. Update the second weight: $$w_2' = -0.2 - (0.1 \times (-0.4)) = -0.2 - (-0.04) = -0.16$$
3. Update the bias: $$b' = 0.1 - (0.1 \times 0.05) = 0.1 - 0.005 = 0.095$$
4. Final JSON result: {"bias":0.095,"weights":[0.48,-0.16]}.

• Python Pattern:

  if len(weights) != len(grad_weights) or lr <= 0:
      raise ValueError("Invalid gradient parameters")
  
  new_weights = [round_value(w - lr * gw) for w, gw in zip(weights, grad_weights)]
  new_bias = round_value(bias - lr * grad_bias)
  
  # Insert 'bias' first to maintain alphabetical order:
  return {
      "bias": new_bias,
      "weights": new_weights
  }
  

• TypeScript Pattern:

  if (weights.length !== grad_weights.length || lr <= 0) {
    throw new Error("Invalid gradient parameters");
  }
  
  const newWeights = weights.map((w, i) =>
    roundValue(w - lr * grad_weights[i]),
  );
  const newBias = roundValue(bias - lr * grad_bias);
  
  // Insert 'bias' before 'weights' to ensure alphabetical key order:
  return {
    bias: newBias,
    weights: newWeights,
  };
  

1. If learning_rate <= 0, print error to stderr and exit with code 1.
2. If weights and grad_weights differ in length, print error to stderr and exit with code 1.
3. Round each updated number to 4 decimal places with round_value / roundValue. If integer (including 0), serialize as an integer.
4. Key Ordering: Output keys in strict alphabetical order ("bias" followed by "weights"):

   { "bias": 0.095, "weights": [0.48, -0.16] }