PANet Paper Walkthrough: When Feature Pyramids Go Bottom-Up | Towards Data Science
I wrote about the FPN (Feature Pyramid Network) architecture [1], which is one of the most influential necks we can apply to a backbone model. FPN was first introduced to enhance the capability of an object. . .
I wrote about the FPN (Feature Pyramid Network) architecture [1], which is one of the most influential necks we can apply to a backbone model. FPN was first introduced to enhance the capability of an object detection model to detect small objects. However, Liuet al. in 2018 found that the information flow of FPN had room for improvement. So, they decided to address this gap by proposing PANet in their research paper titled “Path Aggregation Network for Instance Segmentation” [2], which we are going to discuss in this article. A Little Bit About FPNPANet is built on top of the FPN architecture, so I think it would be a good idea to talk a little bit about how FPN works in advance. In a CNN-based backbone model, the deeper feature maps have different characteristics from the shallower ones. The feature maps from deeper layers tend to have high semantic information, yet it does not have that much amount of spatial information. Conversely, feature maps from shallower layers contain more spatial information but have less semantic information. These facts essentially tell us that we should use the deeper feature maps if we were to predict large objects and use the shallower feature maps for predicting small objects. However, the authors of PANet found that it is not quite appropriate to directly take information from shallower feature maps to detect small objects. This is because these feature maps contain minimal amount of semantic information, which essentially implies that they do not have a good understanding about the content of the image. FPN solves this issue by combining feature maps from the deeper layers and the shallower ones. By doing so, we can basically inject semantic information into the shallower feature maps that they previously didn’t have. What FPN SolvesNow take a look at the illustration in Figure 1 below to better understand this idea. The upward flow on the left is the main backbone model, while the one going downwards on the right (a. k. a. thetop-down pathway) is the FPN. The horizontal arrows that go directly to the FPN part (calledlateral connections) are the tensors to be combined with the one from the deeper layers. In this illustration, the FPN produces three feature maps where the detection head will make the bounding box predictions from. Figure 1. The FPN architecture [3]. The blue borderlines in the illustration above indicate the amount of semantic information contained within the corresponding feature map. You can see in the original backbone that the borderline gets thicker as we get deeper into the network, meaning that the feature map that has the highest semantic information is the one produced by the deepest layer. By using feature map combination mechanism proposed by FPN, we can see that the shallower feature maps are now rich of semantic information which makes them suitable for detecting small objects accurately. What FPN Doesn’t SolveAt this point we might think that FPN is just fine as it is able to allow a model to detect small objects better compared to the plain backbone model. In fact, the authors of PANet still see something that isn’t solved by FPN just yet, which as I’ve mentioned earlier is related to the information flow problem. Now let’s take a look at Figure 2 below. Figure 2. The PANet architecture [2]. In this figure, the component referred to as (a) is the backbone model that has been equipped with FPN, which is basically the same structure I showed you in Figure 1. In FPN, feature mapsP₅,P₄,P₃, andP₂become the basis of the detection head to make predictions, whereP₂has a high semantic information thanks to the flow in the top-down pathway. And here’s the problem FPN doesn’t solve: while FPN effectively enriches feature maps in shallower layers, the deeper feature maps still lack spatial information. In the beginning of this article I said that the deeper and the shallower feature maps in a backbone have their own advantage and drawback, and FPN only solves the drawback of the shallower feature maps while still leaving the problem of the deeper feature maps untouched. Based on this notion, the authors of PANet thought that we could still improve the ability of an object detection model in detecting large objects by injecting spatial information to the deeper feature maps, which also potentially allows the overall accuracy of the detection model to get even higher. It is definitely normal for a deeper layer to lose spatial information due to the downsampling mechanism in the backbone. However, this is actually not solely caused by the downsampling itself. Instead, the long information travel in the stack of convolution layers also causes spatial information to degrade. So, if we cannot omit the downsampling operation within the backbone, we can still minimize this spatial information degradation by creating shortcut paths — just like the skip-connections in ResNet. How PANet Solves the Problem Unsolved by FPNThe authors of PANet proposed a clever idea, where they added a stack of several convolution layers placed on top of the existing FPN alongside the lateral connections (the part referred to as (b) in Figure 2). It might seem counterintuitive at glance — how can adding more layers allow us to get shorter paths instead? In fact, this approach is valid as it really allows the shallower feature maps to deliver spatial information to the deeper layers more seamlessly. Now let’s go back to Figure 2 and pay attention to the dashed arrows. Both arrows essentially denote the flow of theC₂tensor (the backbone feature at the same spatial resolution asP₂andN₂) to the deepest layer. The red one is when we only use the top-down pathway of FPN, while the green one is when we also use thebottom-up path augmentationof PANet. If we don’t utilize PANet, the only way for theC₂tensor to get into the last layerP₅is by traveling up through the entire backbone one by one, which can actually be over 100 layers in total. And as we discussed earlier, this causes degradation in the spatial information. If we compare this with the flow when we use PANet, it essentially allows us to use the path traced by the green arrow. Keep in mind that the number of layers within the top-down and the bottom-up pathways are much less than the one in the backbone, hence reducing the path from 100+ layers to around 10 layers to reachN₅(which is in the same level asP₅). By doing this, we can definitely preserve much more spatial information. And so, by combining FPN and PANet, we get bidirectional information flow that enriches shallower layers with semantic information and deeper layers with spatial information. By the way, in this article we will focus on how we can integrate the (a) and (b) parts in the architecture, which is the core contribution that has been widely adopted in modern architectures. The full PANet paper actually also includes the adaptive pooling mechanism (c), bounding box prediction for object detection (d), and fully-connected feature fusion for segmentation (e), which are beyond our scope here. The Detailed PANet ArchitectureIn FPN, theP₅,P₄,P₃, andP₂tensors will directly be forwarded to the detection head, whereas in PANet, these four tensors will be transferred to the bottom-up path augmentation which then produces theN₅,N₄,N₃, andN₂tensors. TheseNtensors are the ones to be used by the detection head as the basis of the object detection prediction. Generally speaking, the way PANet performs feature combination is similar to FPN yet is done in reversed way. Take a look at Figure 3 below to see the detailed process. Figure 3. An illustration of the building block for bottom-up path augmentation [2]. What we initially do in the above building block is to process theNᵢtensor with a 3×3 convolution. The stride of this conv layer is set to 2 since we want it to reduce the spatial dimension by half. Next, we combine thePᵢ₊₁tensor and the downsampledNᵢtensor using element-wise summation. This combined tensor is then processed by another 3×3 convolution layer. This time we don’t want this layer to reduce the spatial dimension any further, so we need to set the stride to 1. Finally, as all processes above are done, we will have theNᵢ₊₁tensor, which is ready to be processed again with the same procedure until we eventually get all theN₅,N₄,N₃, andN₂tensors. There are actually several things you need to keep in mind here. First, sinceN₂is the very first tensor in the bottom-up path augmentation, there is actually nothing to combine to obtain this tensor. Thus, we can simply perceiveP₂as theN₂tensor itself. Second, here we need to put a ReLU activation function after each convolution layer. In fact, this is different from FPN which completely omits the activation functions. So, later in the implementation, our top-down pathway will not use any activation function at all, yet we will have all the convs in the bottom-up path augmentation equipped with ReLU. And third, all tensors throughout the entire bottom-up path augmentation have exactly 256 channels, meaning that the convolution layers here do not change the number of channels at all. That’s pretty much all about the detailed flow of the PANet architecture. In the next section I am going to demonstrate how to implement it from scratch. PANet From ScratchThe idea of our implementation here is that we will create a dummy backbone with an FPN and place the PANet layers on top of it. As usual, what we need to do first in the code is to import the required modules. See Codeblock 1 below. # Codeblock 1 import torch import torch. nn as nnCNN ImplementationWhat I actually mean by a dummy backbone is a very simple CNN model. In fact theCNNclass in Codeblock 2 below is exactly the same as the one in my FPN article [1]. Just keep in mind that the actual backbone used in the paper is ResNet, but since the focus of this article is PANet, I’ll use a standard CNN instead. (By the way, I actually got a separate article about ResNet. Check that out at the link in reference [4] in case you’re interested to implement FPN on ResNet from scratch. )# Codeblock 2 class CNN(nn. Module): def __init__(self): super(). __init__() self. relu = nn. ReLU() self. maxpool = nn. MaxPool2d(kernel_size=3, stride=2, padding=1) self. conv1 = nn. Conv2d(in_channels=3, out_channels=64, kernel_size=3, padding=1) self. conv2 = nn. Conv2d(in_channels=64, out_channels=256, kernel_size=3, padding=1) self. conv3 = nn. Conv2d(in_channels=256, out_channels=512, kernel_size=3, padding=1) self. conv4 = nn. Conv2d(in_channels=512, out_channels=1024, kernel_size=3, padding=1) self. conv5 = nn. Conv2d(in_channels=1024, out_channels=2048, kernel_size=3, padding=1) def forward(self, x): print(f'original\t: {x. size()}\n') x = self. relu(self. conv1(x)) print(f'after conv1\t: {x. size()}') x = self. maxpool(x) print(f'after pool\t: {x. size()}\n') x = self. relu(self. conv2(x)) print(f'after conv2\t: {x. size()}') x = self. maxpool(x) print(f'after pool (c2)\t: {x. size()}\n') c2 = x. clone() x = self. relu(self. conv3(x)) print(f'after conv3\t: {x. size()}') x = self. maxpool(x) print(f'after pool (c3)\t: {x. size()}\n') c3 = x. clone() x = self. relu(self. conv4(x)) print(f'after conv4\t: {x. size()}') x = self. maxpool(x) print(f'after pool (c4)\t: {x. size()}\n') c4 = x. clone() x = self. relu(self. conv5(x)) print(f'after conv5\t: {x. size()}') c5 = self. maxpool(x) print(f'after pool (c5)\t: {c5. size()}\n') return c2, c3, c4, c5TheCNNclass above returns multiple outputs:c2,c3,c4andc5, wherec5is the tensor from the deepest layer. In order to check if this class works properly, we will test it by passing a tensor through it. The dummy tensor I initialize in Codeblock 3 below has the dimension of 3×224×224, which matches the input shape of the ResNet model. # Codeblock 3 cnn = CNN() x = torch. randn(1, 3, 224, 224) out_cnn = cnn(x) c2, c3, c4, c5 = out_cnnBelow you can see how the backbone model transforms our input tensor throughout the network. Later on, all theCtensors will be used as the input of our FPN. # Codeblock 3 Output original : torch. Size([1, 3, 224, 224]) after conv1 : torch. Size([1, 64, 224, 224]) after pool : torch. Size([1, 64, 112, 112]) after conv2 : torch. Size([1, 256, 112, 112]) after pool (c2) : torch. Size([1, 256, 56, 56]) after conv3 : torch. Size([1, 512, 56, 56]) after pool (c3) : torch. Size([1, 512, 28, 28]) after conv4 : torch. Size([1, 1024, 28, 28]) after pool (c4) : torch. Size([1, 1024, 14, 14]) after conv5 : torch. Size([1, 2048, 14, 14]) after pool (c5) : torch. Size([1, 2048, 7, 7])FPN ImplementationAnd here’s what theFPNclass looks like. I am not going to get very deep into the flow of this network since I’ve covered it thoroughly in my previous article [1]. # Codeblock 4 class FPN(nn. Module): def __init__(self): super(). __init__() self. upsample = nn. Upsample(scale_factor=2, mode='nearest') #(1) self. lateral_c5 = nn. Conv2d(in_channels=2048, out_channels=256, kernel_size=1) self. lateral_c4 = nn. Conv2d(in_channels=1024, out_channels=256, kernel_size=1) self. lateral_c3 = nn. Conv2d(in_channels=512, out_channels=256, kernel_size=1) self. lateral_c2 = nn. Conv2d(in_channels=256, out_channels=256, kernel_size=1) self. smooth_m4 = nn. Conv2d(in_channels=256, out_channels=256, kernel_size=3, padding=1) self. smooth_m3 = nn. Conv2d(in_channels=256, out_channels=256, kernel_size=3, padding=1) self. smooth_m2 = nn. Conv2d(in_channels=256, out_channels=256, kernel_size=3, padding=1) def forward(self, c2, c3, c4, c5): m5 = self. lateral_c5(c5) p5 = m5 m4 = self. upsample(m5) + self. lateral_c4(c4) p4 = self. smooth_m4(m4) m3 = self. upsample(m4) + self. lateral_c3(c3) p3 = self. smooth_m3(m3) m2 = self. upsample(m3) + self. lateral_c2(c2) p2 = self. smooth_m2(m2) return p2, p3, p4, p5For now, let’s just focus on the returnedPtensors which we will use as the input of our PANet. You can see in the output of the code below that all thesePtensors have 256 channels, which I believe is why the authors eventually ended up with this number of channels for the entire PANet structure. # Codeblock 5 fpn = FPN() out_fpn = fpn(c2, c3, c4, c5) p2, p3, p4, p5 = out_fpn print(f'p2: {p2. size()}') print(f'p3: {p3. size()}') print(f'p4: {p4. size()}') print(f'p5: {p5. size()}')# Codeblock 5 Output p2: torch. Size([1, 256, 56, 56]) p3: torch. Size([1, 256, 28, 28]) p4: torch. Size([1, 256, 14, 14]) p5: torch. Size([1, 256, 7, 7])PANet ImplementationAnd finally, it is time to implement the PANet itself. In Codeblock 6 below, what we do first is to initialize thedownsample_n{2,3,4}layers. As the name suggests, these layers are responsible for performing spatial downsampling, which is the reason that we set thestrideparameter of these layers to 2. Next, we initializeconv_{n2down_p3, n3down_p4, n4down_p5}. These three convolution layers are used to process the combined tensors. For instance, the layer with suffixn2down_p3will take the combined downsampledN₂and theP₃tensors for the input. Don’t forget to initialize the ReLU activation function as we will use them after each convolution layer. # Codeblock 6 class PANet(nn. Module): def __init__(self): super(). __init__() self. downsample_n2 = nn. Conv2d(in_channels=256, out_channels=256, kernel_size=3, stride=2, padding=1) self. downsample_n3 = nn. Conv2d(in_channels=256, out_channels=256, kernel_size=3, stride=2, padding=1) self. downsample_n4 = nn. Conv2d(in_channels=256, out_channels=256, kernel_size=3, stride=2, padding=1) self. conv_n2down_p3 = nn. Conv2d(in_channels=256, out_channels=256, kernel_size=3, padding=1) self. conv_n3down_p4 = nn. Conv2d(in_channels=256, out_channels=256, kernel_size=3, padding=1) self. conv_n4down_p5 = nn. Conv2d(in_channels=256, out_channels=256, kernel_size=3, padding=1) self. relu = nn. ReLU() def forward(self, p2, p3, p4, p5): n2 = p2 #(1) print(f'n2\t\t: {n2. size()}\n') ####################################### n2down = self. relu(self. downsample_n2(n2)) #(2) print(f'n2 downsampled\t: {n2down. size()}') n2down_p3 = n2down + p3 #(3) print(f'after sum\t: {n2down_p3. size()}') n3 = self. relu(self. conv_n2down_p3(n2down_p3)) #(4) print(f'n3\t\t: {n3. size()}\n') ####################################### n3down = self. relu(self. downsample_n3(n3)) print(f'n3 downsampled\t: {n3down. size()}') n3down_p4 = n3down + p4 print(f'after sum\t: {n3down_p4. size()}') n4 = self. relu(self. conv_n3down_p4(n3down_p4)) print(f'n4\t\t: {n4. size()}\n') ####################################### n4down = self. relu(self. downsample_n4(n4)) print(f'n4 downsampled\t: {n4down. size()}') n4down_p5 = n4down + p5 print(f'after sum\t: {n4down_p5. size()}') n5 = self. relu(self. conv_n4down_p5(n4down_p5)) print(f'n5\t\t: {n5. size()}') return n2, n3, n4, n5Now let’s move on to theforward()method. You can see here that we accept allPtensors from FPN as the input. As I’ve mentioned earlier, we don’t need to do anything with theP₂tensor since its correspondingNtensor is the very first one in the bottom-up path augmentation (#(1)). Next, at the line marked with#(2)we downsample then2tensor such that its spatial dimension halves. This downsampling process is important since we want then2tensor to have the exact same dimension asp3to allow element-wise summation to be performed (#(3)). Afterwards, this combined tensor is processed further by another convolution layer (#(4)). These steps are repeated several times until we eventually reachn5. Now let’s test our code above by initializing a PANet instance and pass thePtensors we obtained from our previous output through the network. # Codeblock 7 panet = PANet() out_panet = panet(p2, p3, p4, p5) n2, n3, n4, n5 = out_panetAnd below is what the output looks like. It seems like our PANet works properly as it successfully passes the input tensor all the way to the end of the network. It is also seen here that while the spatial dimension gets smaller, the number of feature maps in each step remains the same, which is exactly what we want. # Codeblock 7 Output n2 : torch. Size([1, 256, 56, 56]) n2 downsampled : torch. Size([1, 256, 28, 28]) after sum : torch. Size([1, 256, 28, 28]) n3 : torch. Size([1, 256, 28, 28]) n3 downsampled : torch. Size([1, 256, 14, 14]) after sum : torch. Size([1, 256, 14, 14]) n4 : torch. Size([1, 256, 14, 14]) n4 downsampled : torch. Size([1, 256, 7, 7]) after sum : torch. Size([1, 256, 7, 7]) n5 : torch. Size([1, 256, 7, 7])Everything in a Single PassLastly, let’s try to wrap everything in a single class which I refer to asBackboneNeck. The way to do it is extremely simple, all we need to do in the__init__()method is just to initialize the dummy CNN, FPN, and the PANet we created earlier. Next, in theforward()method, we can just pass the tensors sequentially. Here I also print out the shape of the tensors in the intermediate steps so that you can clearly see how the tensors transform after each part of the network. The outputs of thisBackboneNeckare theNtensros, which are ready to be forwarded to a detection head. # Codeblock 8 class BackboneNeck(nn. Module): def __init__(self): super(). __init__() self. cnn = CNN() self. fpn = FPN() self. panet = PANet() def forward(self, x): c2, c3, c4, c5 = self. cnn(x) print(f'c2: {c2. size()}') print(f'c3: {c3. size()}') print(f'c4: {c4. size()}') print(f'c5: {c5. size()}\n') p2, p3, p4, p5 = self. fpn(c2, c3, c4, c5) print(f'p2: {p2. size()}') print(f'p3: {p3. size()}') print(f'p4: {p4. size()}') print(f'p5: {p5. size()}\n') n2, n3, n4, n5 = self. panet(p2, p3, p4, p5) print(f'n2: {n2. size()}') print(f'n3: {n3. size()}') print(f'n4: {n4. size()}') print(f'n5: {n5. size()}') return n2, n3, n4, n5Now let’s test ourBackboneNeckclass above using the following testing code. # Codeblock 9 backbone_neck = BackboneNeck() x = torch. randn(1, 3, 224, 224) n2, n3, n4, n5 = backbone_neck(x)Here we can see that our dummy RGB image of size 224×224 successfully flows through the entire network. TheCtensors produced by the backbone initially have varying numbers of channels (#(1–4)). Our FPN then normalizes them to a uniform 256 channels while enriching the shallower layers with semantic information (#(5–8)). We further process the resultingPtensors with PANet to inject spatial information to the deeper layers, resulting in theNtensors while maintaining the same spatial dimensions as their correspondingPtensors (#(9–12)). # Codeblock 9 Output original : torch. Size([1, 3, 224, 224]) after conv1 : torch. Size([1, 64, 224, 224]) after pool : torch. Size([1, 64, 112, 112]) after conv2 : torch. Size([1, 256, 112, 112]) after pool (c2) : torch. Size([1, 256, 56, 56]) after conv3 : torch. Size([1, 512, 56, 56]) after pool (c3) : torch. Size([1, 512, 28, 28]) after conv4 : torch. Size([1, 1024, 28, 28]) after pool (c4) : torch. Size([1, 1024, 14, 14]) after conv5 : torch. Size([1, 2048, 14, 14]) after pool (c5) : torch. Size([1, 2048, 7, 7]) c2: torch. Size([1, 256, 56, 56]) #(1) c3: torch. Size([1, 512, 28, 28]) #(2) c4: torch. Size([1, 1024, 14, 14]) #(3) c5: torch. Size([1, 2048, 7, 7]) #(4) p2: torch. Size([1, 256, 56, 56]) #(5) p3: torch. Size([1, 256, 28, 28]) #(6) p4: torch. Size([1, 256, 14, 14]) #(7) p5: torch. Size([1, 256, 7, 7]) #(8) n2: torch. Size([1, 256, 56, 56]) #(9) n3: torch. Size([1, 256, 28, 28]) #(10) n4: torch. Size([1, 256, 14, 14]) #(11) n5: torch. Size([1, 256, 7, 7]) #(12)EndingAnd that’s everything about how PANet works, especially regarding the way it enriches spatial information in the deeper layers. There are actually several other additional components of PANet I haven’t covered in this article, i. e. , the (c), (d), and (e) components in Figure 2, which I think I’ll save it for my upcoming articles. Thank you very much for reading. You can also find the code used in this article in my GitHub repo [5]. Please let me know if there are mistakes in the code or in my explanation. See ya in my next article!References[1]Muhammad Ardi. FPN Paper Walkthrough: Leveraging the Internal Pyramid. https://towardsdatascience. com/fpn-paper-walkthrough-leveraging-the-internal-pyramid/[Accessed July 3, 2026]. [2] Shu Liuet al. Path Aggregation Network for Instance Segmentation. Arxiv. https://arxiv. org/abs/1803. 01534[Accessed October 8, 2025]. [3] Tsung-Yi Linet al. Feature Pyramid Networks for Object Detection. Arxiv. https://arxiv. org/abs/1612. 03144[Accessed October 8, 2025]. [4]Muhammad Ardi. Paper Walkthrough: Residual Network (ResNet). Python in Plain English. https://medium. com/python-in-plain-english/paper-walkthrough-residual-network-resnet-62af58d1c521[Accessed October 8, 2025]. [5] MuhammadArdiPutra. PANet. GitHub. https://github. com/MuhammadArdiPutra/medium_articles/blob/main/Deep%20Learning%20From%20Scratch/PANet. ipynb[Accessed October 8, 2025].