How a Neuron Works

Interactive Neuron

1. Inputs

2. Weights

3. Sum

4. Bias

5. Activation

6. Output

Ready

Click "Run Animation" to see how a neuron computes its output

Speed:

Adjust Inputs

Input 1 (x₁): 0.5

Input 2 (x₂): -0.3

Input 3 (x₃): 0.8

Understanding the Neuron

Think of it like voting: Each input is a voter with a different level of influence (weight). The neuron tallies the weighted votes, adds its own bias, then decides whether to "fire" based on the total.

Step 1: Inputs (x₁, x₂, x₃)

The neuron receives multiple input values. In CartPole, these are the state values: cart position, velocity, pole angle, and angular velocity.

Step 2: Weights (w₁, w₂, w₃)

Each input has an associated weight that determines its importance. Positive weights amplify, negative weights suppress. These weights are what the network learns during training.

Step 3: Weighted Sum

Multiply each input by its weight and sum them all:

sum = x₁×w₁ + x₂×w₂ + x₃×w₃

Step 4: Add Bias

The bias shifts the activation threshold. It lets the neuron fire even when inputs are zero, or require stronger inputs to fire.

z = sum + bias

Step 5: Activation Function (ReLU)

ReLU (Rectified Linear Unit) is simple: if the value is positive, keep it; if negative, output zero.

                    def relu(z):
                        return max(0, z)
                

Why ReLU? It introduces non-linearity, allowing the network to learn complex patterns. Without it, stacking layers would just be matrix multiplication - no better than a single layer!

Step 6: Output

The final output can be passed to the next layer of neurons, or used directly as the network's prediction.

The Full Equation

output = ReLU(x₁×w₁ + x₂×w₂ + x₃×w₃ + bias)

In Python

                    import numpy as np

def neuron(inputs, weights, bias):
    # Weighted sum
    z = np.dot(inputs, weights) + bias
    # ReLU activation
    return max(0, z)
                

Activation Functions Explained

Activation functions are the "decision makers" of neurons. They determine whether and how strongly a neuron should fire based on its input.

Why do we need them? Without activation functions, a neural network is just matrix multiplication - no matter how many layers you stack, it can only learn linear relationships. Activation functions add non-linearity, enabling networks to learn complex patterns.

The Problem Without Activation Functions

Without Activation (Linear Only)

Stacking linear layers just creates another linear function:

W₁x → W₂(W₁x) = (W₂W₁)x

Still just one linear transformation! Can only learn straight lines.

With Activation (Non-Linear)

Non-linearity breaks the chain, enabling complex functions:

W₁x → σ(W₁x) → W₂σ(W₁x)

Now can learn curves, boundaries, and complex patterns!

Interactive Comparison

Input (z): 0.0

ReLU

0.00

Sigmoid

0.50

Tanh

0.00

Leaky ReLU

0.00

Softplus

0.69

Activation Functions Reference

ReLU (Rectified Linear Unit) Recommended

f(x) = max(0, x)

Output range: [0, ∞)

The most popular activation for hidden layers. Simple, fast, and effective.

def relu(x):
    return max(0, x)
                    

Pros

Fast computation
No vanishing gradient (positive)
Sparse activation

Cons

"Dead neurons" problem
Not zero-centered
Unbounded output

Sigmoid (Logistic)

f(x) = 1 / (1 + e⁻ˣ)

Output range: (0, 1)

Squashes input to probability-like output. Used in output layer for binary classification.

def sigmoid(x):
    return 1 / (1 + np.exp(-x))
                    

Pros

Output is bounded
Clear probabilistic interpretation
Smooth gradient

Cons

Vanishing gradients
Not zero-centered
Computationally expensive

Tanh (Hyperbolic Tangent)

f(x) = (eˣ - e⁻ˣ) / (eˣ + e⁻ˣ)

Output range: (-1, 1)

Zero-centered version of sigmoid. Better for hidden layers than sigmoid.

def tanh(x):
    return np.tanh(x)
                    

Pros

Zero-centered
Bounded output
Stronger gradients than sigmoid

Cons

Still has vanishing gradients
Computationally expensive
Saturates at extremes

Leaky ReLU

f(x) = max(αx, x), α = 0.01

Output range: (-∞, ∞)

Fixes the "dead neuron" problem by allowing small negative values.

def leaky_relu(x, alpha=0.01):
    return x if x > 0 else alpha * x
                    

Pros

No dead neurons
Fast computation
Can learn negative values

Cons

Extra hyperparameter (α)
Not zero-centered
Unbounded output

Softplus

f(x) = ln(1 + eˣ)

Output range: (0, ∞)

Smooth approximation of ReLU. Differentiable everywhere.

def softplus(x):
    return np.log(1 + np.exp(x))
                    

Pros

Smooth (no kink)
Always differentiable
Never truly zero

Cons

Computationally expensive
Not widely used
ReLU often works better

Softmax (Output Layer)

f(xᵢ) = eˣⁱ / Σeˣʲ

Output range: (0, 1), sums to 1

Converts scores to probabilities. Used for multi-class classification output.

def softmax(x):
    exp_x = np.exp(x - np.max(x))
    return exp_x / exp_x.sum()
                    

Pros

Probabilities sum to 1
Differentiable
Standard for classification

Cons

Only for output layer
Computationally expensive
Can be numerically unstable

When to Use Each Activation

Layer Type	Recommended	Alternative	Avoid
Hidden Layers	ReLU	Leaky ReLU, ELU	Sigmoid (vanishing gradients)
Binary Classification (Output)	Sigmoid	-	ReLU (unbounded)
Multi-Class Classification (Output)	Softmax	-	Sigmoid (doesn't sum to 1)
Regression (Output)	Linear (none)	ReLU (if non-negative)	Sigmoid/Tanh (bounded)
RNN/LSTM Gates	Sigmoid, Tanh	-	ReLU (exploding activations)
Q-Network (DQN)	ReLU (hidden)	Leaky ReLU	Sigmoid (limits Q range)

The "Dead Neuron" Problem

What happens: If a neuron's input is always negative, ReLU outputs zero. With zero output, the gradient is zero, so weights never update. The neuron is "dead" - it will never recover and learn anything!

Causes

Learning rate too high (weights pushed negative)
Poor weight initialization
Bad luck during training
Large negative bias

Solutions

Use Leaky ReLU or ELU instead
Lower learning rate
Use He initialization for weights
Use batch normalization

Key Takeaways

Rule of Thumb:

Hidden layers: Start with ReLU. If you see dead neurons, try Leaky ReLU.
Output layer: Match to your task (Sigmoid for binary, Softmax for multi-class, Linear for regression).
When in doubt: ReLU is almost always a safe choice for hidden layers.

How a Single Neuron Works

Interactive Neuron

Adjust Inputs

Understanding the Neuron

Step 1: Inputs (x₁, x₂, x₃)

Step 2: Weights (w₁, w₂, w₃)

Step 3: Weighted Sum

Step 4: Add Bias

Step 5: Activation Function (ReLU)

Step 6: Output

The Full Equation

In Python

Activation Functions Explained

The Problem Without Activation Functions

Without Activation (Linear Only)

With Activation (Non-Linear)

Interactive Comparison

Activation Functions Reference

ReLU (Rectified Linear Unit) Recommended

Sigmoid (Logistic)

Tanh (Hyperbolic Tangent)

Leaky ReLU

Softplus

Softmax (Output Layer)

When to Use Each Activation

The "Dead Neuron" Problem

Causes

Solutions

Key Takeaways