⚡ An activation function is a mathematical formula inside each neuron of a neural network that decides how strongly to pass a signal forward. Without it, a network with 100 layers behaves exactly like a network with 1 layer — it can only learn straight-line relationships, not the complex patterns that make AI useful.

Category: Deep Learning · Difficulty: Intermediate · Last updated: 15 May 2026 · 5 min read

What is Activation Function?

Every neuron in a neural network receives signals from the previous layer, multiplies them by weights, adds a bias — and then passes the result through an activation function before sending it to the next layer. That final step, the activation function, is what gives neural networks their power.

Without activation functions, stacking layers does nothing. Mathematically, a sequence of linear operations collapses into a single linear operation no matter how many layers you add. Activation functions introduce non-linearity — the ability to model curves, edges, complex boundaries — which is exactly what is needed to recognise faces, understand language, and detect tumours.

How Activation Function works ?

  1. A neuron receives inputs — numbers from the previous layer multiplied by connection weights.
  2. It sums all the weighted inputs together (plus a bias value).
  3. It passes that sum through the activation function.
  4. The output of the activation function is what gets sent to the next layer.
  5. The choice of activation function determines what patterns that neuron can represent.
  6. Training adjusts the weights; the activation function shapes what patterns are possible.

COMMON TYPES

  • ReLU (Rectified Linear Unit) — outputs zero for negative inputs, the input itself for positive. Fast, simple, default choice for hidden layers. Used in most modern deep networks.
  • Sigmoid — squashes any input into a range between 0 and 1. Used in output layers for binary classification (yes/no decisions).
  • Softmax — converts a vector of numbers into a probability distribution summing to 1. Used in output layers for multi-class classification.
  • Tanh — outputs between -1 and 1. Centred at zero, sometimes better than sigmoid for hidden layers. Mostly replaced by ReLU today.
  • GELU — smoother version of ReLU used in transformers (ChatGPT, BERT). Allows small negative values unlike standard ReLU.

When to use Activation Function (and when not to)

✅ Good fit

  • ReLU for hidden layers in most deep learning models
  • Sigmoid for binary yes/no output layers
  • Softmax for multi-class classification output layers
  • GELU for transformer architectures

❌ Bad fit

  • Sigmoid and tanh in deep hidden layers — they cause the vanishing gradient problem, slowing training
  • No activation function — collapses all layers into one, losing all expressive power
  • Using the wrong output activation — sigmoid for multi-class or softmax for binary produces incorrect probability outputs

Real-world examples

  • Image classifiers like ResNet use ReLU throughout hidden layers and softmax at the output to assign probabilities to 1,000 categories.
  • ChatGPT’s transformer layers use GELU activation functions throughout — a smoother variant of ReLU that performs better at scale.
  • A spam filter’s output layer uses sigmoid to output a single number between 0 and 1 representing “probability this is spam.”

Common pitfalls

  • Dying ReLU — neurons that always output zero because inputs are always negative. Fix: use Leaky ReLU or initialise weights carefully.
  • Vanishing gradient — sigmoid and tanh squash values so small that gradients near-vanish in deep networks, preventing learning. Fix: use ReLU in hidden layers.
  • Wrong activation for the task — sigmoid for multi-class classification gives wrong probability outputs. Always match activation to the task type.
  • Treating activation choice as unimportant — for most tasks ReLU works, but for transformers, GELU makes a measurable difference.

Frequently asked questions

QUESTION 1 What is an activation function in simple terms?

ANSWER 1 An activation function is the decision-maker inside each neuron. It takes the incoming signal, applies a mathematical formula, and decides how strongly to pass the signal forward. Without it, a neural network with 100 layers would behave exactly like one with 1 layer.

QUESTION 2 What is ReLU and why is it so popular?

ANSWER 2 ReLU outputs zero for any negative input and the input itself for any positive value. It is popular because it is fast to compute, rarely causes the vanishing gradient problem, and works well in deep networks. It is the default choice for hidden layers in most modern neural networks.

QUESTION 3 What is the difference between sigmoid and softmax?

ANSWER 3 Sigmoid outputs a number between 0 and 1 — used for binary classification (yes or no). Softmax outputs a probability distribution that sums to 1 — used for multi-class classification (which of these 10 categories?).

QUESTION 4 What happens without an activation function?

ANSWER 4 Without activation functions, no matter how many layers you stack, the network only learns linear relationships. It cannot model curves, edges, or patterns. Activation functions are what give neural networks the ability to learn complex, non-linear patterns.

📬 Get one concept + one use case every Tuesday. Join the newsletter →