Non-linear Activation Functions

In the previous stage, you calculated the affine combination of a neuron: $z = \sum w_i x_i + b$. However, there is a fundamental mathematical limitation: linear combinations can only produce straight lines or flat planes.

If you stack multiple layers of purely linear neurons, combining linear operations always results in another linear operation. No purely linear network, regardless of depth, can solve non-linear problems such as the XOR logic gate or circular clusters.

To grant neural networks the expressive power to learn intricate curves, shapes, and complex decision boundaries, the linear sum $z$ must pass through a non-linear activation functionA mathematical function applied to a neuron's output to introduce non-linearityView official documentation.

Inputs (x) ──> [ Weighted Sum: z = Σ w·x + b ] ──> [ Activation: f(z) ] ──> Output (y)

You will implement the two most influential activations in deep learning history:

1. Sigmoid ($\sigma$)

Squashes any real number from the infinite range $(-\infty, +\infty)$ into the bounded interval $(0, 1)$. This makes it ideal for modeling probabilities:

$$\sigma(z) = \frac{1}{1 + e^{-z}}$$

Key intuitive behavior:

• At $z = 0$, the neuron is in a neutral state: $e^{-0} = 1$, yielding $\sigma(0) = \frac{1}{1 + 1} = 0.5$ (50% probability).
• For positive values (e.g. $z = 2.0$), $e^{-2} \approx 0.1353$, and $\sigma(2.0) = \frac{1}{1 + 0.1353} \approx 0.8808$.
• For negative values (e.g. $z = -2.0$), $e^{2} \approx 7.3891$, and $\sigma(-2.0) = \frac{1}{1 + 7.3891} \approx 0.1192$.

2. ReLU (Rectified Linear Unit)

The workhorse of modern deep architectures due to its extreme computational efficiency and resistance to vanishing gradients:

$$\text{ReLU}(z) = \max(0, z)$$

• If $z \ge 0$, it returns $z$ directly (signal passes through).
• If $z < 0$, it returns $0$ (the neuron is deactivated).

When standard input receives "op": "activation", your program must process:

• function: activation name ("sigmoid" or "relu")
• value: floating point number $z$

Examples

Example A (Sigmoid):

{ "op": "activation", "function": "sigmoid", "value": 2.0 }

Output:

{ "output": 0.8808 }

Example B (ReLU):

{ "op": "activation", "function": "relu", "value": -3.2 }

Output:

{ "output": 0 }

• In Python: Use the built-in math module:

  import math
  
  if fn == "relu":
      out = max(0.0, float(value))
  elif fn == "sigmoid":
      out = 1.0 / (1.0 + math.exp(-float(value)))
  

• In TypeScript: Use the global Math object:

  if (fn === "relu") {
    return { output: roundValue(Math.max(0, value)) };
  }
  if (fn === "sigmoid") {
    return { output: roundValue(1 / (1 + Math.exp(-value))) };
  }
  

Common Pitfalls

• Do not drop the minus sign in sigmoid's exponent: it is $e^{-z}$, not $e^{z}$.
• For ReLU, do not confuse with absolute value abs(z): negative numbers must become 0, not positive numbers.

1. Allowed Functions: If function is neither "sigmoid" nor "relu", write error to stderr and exit with code 1.
2. Numeric Precision: Round the result to 4 decimal places. If the rounded value is an integer (such as 0 or 3), serialize it as an integer in JSON.
3. Compact Output: Print only the JSON object terminated by a newline:

   { "output": 0.5 }