Residual learning is a concept in machine learning, particularly in the context of deep neural networks, that aims to make the training process more efficient and the learned features more expressive. It was introduced in the paper “Deep Residual Learning for Image Recognition” by Kaiming He et al. in 2015.

Traditional Neural Networks vs. Residual Networks

In traditional deep neural networks, each layer learns a new representation of the data. As the depth of these networks increases, the optimization of the loss function can become challenging. Very deep networks can suffer from vanishing or exploding gradients, making them hard to train.

In residual learning, instead of trying to learn an underlying mapping �(�)H(x) directly, the neural network learns the residual �(�)=�(�)−�F(x)=H(x)−x, and then recovers �(�)H(x) as �(�)+�F(x)+x.

How it Works

The idea is to add “shortcut connections” that skip one or more layers in a neural network:

Output=�(�,�)+�Output=F(x,W)+x

Here, �(�,�)F(x,W) represents the transformations learned by the network, and �x is the original input. �(�,�)+�F(x,W)+x is the final output of the residual block, where �(�,�)F(x,W) is basically �(�)−�H(x)−x.

These shortcut connections are implemented as identity mappings, meaning they do not have any learnable parameters. They enable the gradient to bypass one or more layers, which helps in alleviating the vanishing and exploding gradient problems and enables the training of deeper networks.

Advantages

1. Easier Optimization: Residual networks are easier to optimize compared to traditional deep networks.
2. Deeper Networks: They allow for the training of much deeper networks, often with hundreds or thousands of layers.
3. Better Performance: Generally, residual networks outperform their non-residual counterparts on a variety of tasks.

Applications

Residual learning has found applications in numerous domains including:

• Image Classification
• Object Detection
• Natural Language Processing
• Reinforcement Learning