# ResNet Architecture Summary ## 1. What This Paper Actually Does Before this paper, it was widely believed that making neural networks deeper would always improve their ability to understand complex data like images. However, researchers encountered a surprising problem: beyond a certain depth, adding more layers actually made the network perform *worse*, even on the data it was trained on. This wasn't due to the network memorizing the training data (overfitting), but because it became incredibly difficult to train effectively, struggling to even learn simple "do-nothing" transformations. This paper introduced a groundbreaking solution called "residual learning." Instead of asking each block of layers to learn a completely new representation, it proposed that these layers should instead learn a *small adjustment* to their input, which is then simply added back to the original input using a direct "shortcut connection." This clever trick made it much easier for the network to learn, especially when an optimal layer should ideally pass information through unchanged. The impact was profound: it enabled the successful training of neural networks with hundreds or even over a thousand layers, leading to significant leaps in image recognition accuracy and establishing a fundamental building block for nearly all advanced deep learning models today. ## 2. The Mechanism ### Briefing Section 2: The Mechanism The core of the Deep Residual Learning framework is a reformulation of what the layers in a deep network are asked to learn. Instead of learning a direct mapping from input to output, the network is trained to learn a residual-the difference between the desired output and the input. This is achieved through a simple but powerful architectural element: the shortcut connection. #### 2.1 Residual Learning Formulation The mechanism is motivated by the degradation problem, where deeper networks show higher training error than their shallower counterparts. This suggests that it is difficult for standard optimizers to learn an identity mapping (where the output is identical to the input) through a stack of non-linear layers, even if that is the optimal solution for the added layers. To address this, the framework reframes the learning objective. Consider a block of layers that we want to learn an underlying mapping, denoted as `H(x)`. Instead of learning `H(x)` directly, the layers are tasked with learning a residual function `F(x)`, defined as: `F(x) := H(x) - x` Here, `x` represents the input to the block of layers, and `H(x)` is the desired output mapping for that block. By rearranging this definition, the original desired mapping `H(x)` is recovered by adding the input `x` back to the output of the residual function: `H(x) = F(x) + x` The hypothesis is that it is easier for an optimizer to learn the residual `F(x)` than the original mapping `H(x)` (el-44). In the extreme case where an identity mapping is optimal (`H(x) = x`), the optimizer can simply drive the weights of the layers learning `F(x)` to zero. This is significantly easier than fitting an identity function through a complex stack of non-linear transformations like convolutional layers and ReLUs. #### 2.2 The Residual Block and Shortcut Connection This `F(x) + x` formulation is implemented using a "residual block" containing a "shortcut connection," as conceptually shown in the paper's Figure 2. A residual block consists of two paths: 1. A main path containing a few weighted layers (e.g., convolutional, Batch Normalization, ReLU) that learn the residual mapping `F(x)`. 2. A shortcut path that bypasses these layers and carries the input `x` forward. The outputs of these two paths are then combined through element-wise addition. The output of the block, `y`, is formally defined as: **`y = F(x, {W_i}) + x`** (1) Here, we decode the symbols: * `y` is the output vector of the residual block. * `x` is the input vector to the block. * `F(x, {W_i})` represents the residual mapping learned by the weighted layers in the main path. * `{W_i}` denotes the set of weights and biases associated with these layers. For a typical two-layer block, `F` would be of the form `W_2 * σ(BN(W_1 * x))`, where `W_1` and `W_2` are weight matrices, `BN` is Batch Normalization, and `σ` is the ReLU activation function (el-53). This identity shortcut `+ x` is the key component. It requires no additional parameters and adds negligible computational cost. Because it provides a direct, uninterrupted path for information and gradients to flow, it greatly simplifies the optimization of very deep networks. The entire architecture can be trained end-to-end with standard SGD and backpropagation. #### 2.3 Handling Dimension Mismatches The element-wise addition in Equation (1) is only possible if the input `x` and the output of the residual function `F(x)` have the same dimensions (i.e., same height, width, and number of channels). However, in deep CNNs, it is common for convolutional layers to use a stride greater than one, which reduces spatial dimensions, or to increase the number of feature maps (channels). To handle these dimension mismatches, the identity shortcut is replaced with a linear projection shortcut. The formulation becomes: **`y = F(x, {W_i}) + W_s * x`** (2) The new symbol is: * `W_s` is a projection matrix, implemented as a 1x1 convolution. Its purpose is solely to match the dimensions of `x` to the dimensions of `F(x)`. For example, if a block halves the spatial resolution and doubles the number of channels, `W_s` would be a 1x1 convolution with a stride of 2 and twice the number of output channels as input channels. This projection shortcut introduces new parameters but is only used when necessary to align dimensions. The paper's experiments show that parameter-free identity shortcuts (Equation 1) are the most effective and are sufficient to solve the degradation problem, with projection shortcuts serving as a pragmatic solution for changes in dimensionality. ## 3. What You Need To Already Know Here is a dependency-ordered list of concepts foundational to understanding the paper "Deep Residual Learning for Image Recognition": ### 1. Convolutional Neural Networks (CNNs) 1) **Problem**: Standard fully-connected neural networks (MLPs) are inefficient for high-dimensional data like images. They have a massive number of parameters, leading to overfitting, and they do not account for the spatial structure (e.g., locality of pixels) in images. 2) **Solution**: CNNs use specialized layers. Convolutional layers apply learnable filters across the image, sharing weights to detect features regardless of their location. Pooling layers downsample the feature maps, making the representation more robust to small translations. This creates a hierarchy of increasingly complex spatial features. 3) **Usage in this paper**: The ResNet architecture is a very deep Convolutional Neural Network. It is built from stacks of convolutional layers (with 3x3 and 1x1 filters), batch normalization, and ReLU activations, designed for the task of image recognition. ### 2. Deep Neural Networks 1) **Problem**: Shallow neural networks have a limited capacity to represent complex functions. To solve challenging tasks like image recognition, models need to learn a rich hierarchy of features, from simple edges to complex objects. 2) **Solution**: By stacking many layers, a deep neural network can learn features at various levels of abstraction. Each layer learns to represent the features from the previous layer in a more abstract way, increasing the model's expressive power. 3) **Usage in this paper**: The entire paper is motivated by the desire to train *deeper* networks. The authors push the depth to unprecedented levels (152 layers and even over 1000 layers) to show that their residual learning framework overcomes the barriers that previously prevented such deep models from being trained effectively. ### 3. Backpropagation 1) **Problem**: To train a neural network, we need to calculate the gradient of a loss function with respect to every weight in the network. For a deep network with millions of parameters, doing this naively is computationally intractable. 2) **Solution**: Backpropagation is an efficient algorithm for computing these gradients. It uses the chain rule of calculus to iteratively propagate the gradient from the final layer backward through the network, layer by layer, calculating the gradient for each weight along the way. 3) **Usage in this paper**: Backpropagation is the fundamental algorithm used to train all the ResNet models. The paper confirms that the networks can be trained end-to-end by SGD with backpropagation. ### 4. Stochastic Gradient Descent (SGD) 1) **Problem**: Calculating the gradient of the loss function using the entire training dataset (batch gradient descent) is very slow and memory-intensive for large datasets. It can also get stuck in sharp local minima. 2) **Solution**: SGD approximates the true gradient by computing it on a small, random subset of the data called a mini-batch. This is much faster, requires less memory, and the noise introduced by the mini-batch sampling can help the optimizer escape local minima and find better solutions. 3) **Usage in this paper**: All models in the paper are trained using SGD with a momentum term. The mini-batch size is specified as 256 for ImageNet and 128 for CIFAR-10. ### 5. Activation Functions (e.g., ReLU) 1) **Problem**: Traditional activation functions like sigmoid and tanh suffer from the "vanishing gradient problem" in deep networks. Their gradients approach zero for large positive or negative inputs, which means that during backpropagation, the gradient signal can become too small to effectively update the weights in earlier layers, stalling the training process. 2) **Solution**: The Rectified Linear Unit (ReLU), defined as f(x) = max(0, x), is a non-saturating activation function. Its gradient is 1 for all positive inputs, which helps maintain a strong gradient signal during backpropagation, leading to faster and more effective training of deep networks. 3) **Usage in this paper**: ReLU is used as the non-linear activation function (denoted by σ) within the residual building blocks, typically after a batch normalization layer. ### 6. Batch Normalization 1) **Problem**: During training, the distribution of each layer's inputs changes as the parameters of the preceding layers are updated. This phenomenon, called "internal covariate shift," slows down training because the network has to constantly adapt to these changing distributions. It also makes the network highly sensitive to weight initialization. 2) **Solution**: Batch Normalization normalizes the output of a previous layer before it is fed to the next. For each mini-batch, it standardizes the activations to have zero mean and unit variance, and then applies a learnable scale and shift. This stabilizes the input distributions, allowing for higher learning rates and making the network less sensitive to initialization. 3) **Usage in this paper**: Batch Normalization is a critical component of the ResNet architecture. It is applied right after each convolution and before the ReLU activation. The authors note that BN helps address the vanishing gradient problem, allowing them to focus on the separate degradation problem. ### 7. Vanishing/Exploding Gradients 1) **Problem**: In very deep networks, as the gradient is backpropagated from the output layer to the input layer, it is repeatedly multiplied by the weights of each layer. If these weights are small, the gradient can shrink exponentially (vanish), preventing early layers from learning. If the weights are large, the gradient can grow exponentially (explode), causing unstable training. 2) **Solution**: This problem is addressed by a combination of techniques: careful weight initialization (e.g., He or Xavier initialization), non-saturating activation functions (e.g., ReLU), and intermediate normalization layers (e.g., Batch Normalization). Shortcut connections also provide a more direct path for the gradient to flow. 3) **Usage in this paper**: The paper argues that the degradation problem they address is distinct from the vanishing gradient problem, which they state has been 'largely addressed' by techniques like Batch Normalization, which they use extensively. ### 8. Identity Mapping / Shortcut Connections 1) **Problem**: The degradation problem shows that it is difficult for a stack of non-linear layers to learn an identity mapping (i.e., a function where the output is simply the input). If a shallower network is optimal, a deeper network should be able to perform at least as well by learning identity functions for the extra layers, but in practice, optimizers fail to find this solution. 2) **Solution**: Shortcut (or skip) connections provide a direct path for data to bypass one or more layers. An identity shortcut adds the input `x` to the output of the layers `F(x)`, resulting in `F(x) + x`. If the identity mapping is optimal, the network can easily achieve this by learning to make `F(x)` zero, which is easier than fitting an identity function with non-linear layers. 3) **Usage in this paper**: This is the core mechanism of the proposed residual learning framework. Every residual block contains an identity shortcut connection that adds the block's input to its output, enabling the successful training of extremely deep networks.