Understanding Convolutional Neural Networks

Prerequisites

This article assumes you have:

• A basic understanding of deep learning, particularly feedforward neural networks.
• Familiarity with high-school level mathematics.

Introduction

If you've spent any time learning about machine learning, you've almost certainly come across Artificial Neural Networks (ANNs). They're often described as being inspired by the human brain. While that comparison isn't entirely inaccurate, it's more helpful to think of neural networks as mathematical models rather than biological simulations.

At their core, neural networks are systems designed to learn patterns from data.

An ANN is made up of many interconnected computational units, commonly referred to as neurons, which are arranged into layers. Each neuron performs a mathematical operation on its inputs before passing the result to the next layer. Although each individual computation is relatively simple, the combined effect of thousands or even millions of these operations allows a network to model remarkably complex relationships.

A typical feedforward neural network looks something like this:

When we train a neural network, we provide it with input data, usually represented as a multidimensional vector. This data is passed through one or more hidden layers before producing an output. During training, the network continually adjusts the weights of its connections in an effort to reduce prediction error. Over many iterations, these adjustments enable the model to learn increasingly useful representations of the data.

When several hidden layers are stacked together, the model is referred to as a deep neural network, and the process of training these models is commonly known as deep learning.

As neural networks become larger and deeper, the number of trainable parameters increases. This naturally leads to greater computational and memory requirements. These challenges become even more significant when working with high-dimensional data such as images.

Fortunately, machine learning is not limited to a single type of data. Depending on the problem, we might work with vectors, sequences, graphs, or multidimensional arrays. One of the reasons machine learning is so versatile is that the same underlying principles can be applied to many different forms of data.

In this article, however, we'll focus on one particular representation: grid-structured data.

Images are perhaps the most familiar example of grid-structured data, and understanding how computers represent images naturally leads us to one of the most important architectures in modern deep learning, the Convolutional Neural Network (CNN).

Images As Numerical Data

To us, an image is a photograph, a drawing, or perhaps even the face of someone we know.

To a computer, an image is simply a collection of numbers.

Every image is represented as a grid of pixels, where each pixel stores one or more numerical values.

For a grayscale image, each pixel contains a single intensity value, typically ranging from 0, representing black, to 255, representing white. Values between these extremes correspond to different shades of gray.

Color images follow exactly the same principle, except that each pixel stores three intensity values corresponding to the Red, Green, and Blue (RGB) color channels. These three matrices are combined to produce the color image we see. Mathematically, an RGB image can therefore be viewed as a three-dimensional array, often referred to as a tensor.

This method of representing information is not unique to images. Audio spectrograms, medical scans, satellite imagery, and many other forms of data can also be represented as multidimensional arrays. This shared structure allows similar machine learning techniques to be applied across many different domains.

The Problem With Fully Connected Networks

At first glance, it might seem that a traditional feedforward neural network should have no difficulty processing images. After all, an image is simply another collection of numbers.

The problem lies in how a fully connected network processes those numbers.

Every input feature is connected to every neuron in the following layer. While this approach works well for relatively small inputs, it becomes increasingly impractical as image size grows.

Consider a grayscale image with a resolution of 256 × 256. Such an image contains 65,536 input values. An RGB image of the same resolution contains 196,608 values.

Now imagine connecting every one of those values to just 1,000 neurons in the first hidden layer. Before the network has learned a single feature, it already contains nearly 200 million trainable parameters.

That is an enormous amount of computation.

Unfortunately, computation is only part of the problem.

By flattening a two-dimensional image into a one-dimensional vector, we discard the image's natural spatial structure. Two neighboring pixels, which are often closely related, become nothing more than two values somewhere within a long list of numbers. The network must then learn these spatial relationships from the training data alone, requiring even more parameters and significantly larger datasets.

Although fully connected neural networks can process images, they are simply not designed for this type of data.

This naturally raises an important question

Why Not Just Build A Larger Neural Network?

Computational Complexity

The most obvious limitation is computational cost.

Increasing the number of hidden layers or neurons certainly gives a network more capacity to learn complex functions. However, it also increases the number of trainable parameters. More parameters require more memory, longer training times, and greater computational resources.

For high-resolution images, these requirements quickly become impractical, even on modern hardware.

Overfitting

The second limitation is overfitting.

A model overfits when it learns the training data so closely that it begins memorizing specific examples instead of learning the underlying patterns. As a result, the model performs very well on the training data but struggles when presented with new, unseen data.

Large fully connected networks are especially susceptible to overfitting because they contain such a large number of trainable parameters. Unless an equally large amount of training data is available, the model can begin fitting random noise instead of meaningful features.

A New Approach: Convolutional Neural Networks

 This problem required a different solution, or perhaps more accurately, a different approach. That approach came in the form of Convolutional Neural Networks (CNNs).

Although CNNs are classified as a type of neural network, they are not simply traditional neural networks made larger or deeper. Their key difference lies in the way they process information before it reaches the fully connected layers.

At a high level, a CNN introduces a series of specialized operations that extract meaningful features from an image before passing that information to a traditional neural network component. These operations act as a form of automated feature extraction, allowing the network to identify important patterns such as edges, textures, and shapes.

But, why are CNNs considered a type of neural network?

 The answer lies in their architecture.

A CNN still follows the fundamental principles of neural networks. It contains layers of interconnected computational units, it has trainable parameters, and it learns by adjusting these parameters during training. The difference is that some of these layers are specifically designed to process spatial information through operations such as convolution and pooling.

While describing these early stages as 'image processing layers' provides an intuitive explanation, they are not fixed image-processing algorithms. Instead, they are learnable components of the neural network itself. During training, the CNN learns which features are useful for solving the given task.

These feature extraction layers form the foundation that allows the network to understand the structure of an image before making predictions through later layers. In this way, CNNs combine traditional neural network learning with specialized mechanisms designed for handling visual data.

What Does A CNN Architecture Look Like?

Now that we understand why Convolutional Neural Networks were introduced, the next question is: what does a CNN actually look like?

A CNN is built by combining different types of layers, each designed to perform a specific role in processing and understanding image data. The three fundamental layer types found in most CNN architectures are:

• Convolutional Layer
• Pooling Layer
• Fully Connected Layer

These layers can be arranged in different combinations and configurations depending on the complexity of the problem and the type of data being processed. Some architectures may contain only a few layers, while others may contain hundreds or even thousands of them.

Although CNN architectures can vary significantly, their underlying principle remains the same: they are designed to efficiently learn patterns from grid-structured data, particularly images. When trained effectively, CNNs have demonstrated remarkable performance across a wide range of computer vision tasks.

A simplified CNN architecture used for MNIST digit classification can be represented as follows:

Understanding The Layers of A CNN

Now that we have a basic understanding of the overall CNN architecture, let's take a closer look at each of the individual layers and understand the role they play in processing image data.

The Input Layer

As discussed earlier, the input layer is where we provide the model with the images it needs to learn from.

Unlike traditional fully connected neural networks, we do not need to flatten the image into a one-dimensional vector before passing it into a CNN. Flattening involves converting a multidimensional structure into a single row or column of values, which removes the explicit spatial arrangement of the data.

For CNNs, this spatial structure is extremely important. Instead of converting an image into a long list of pixel values, we provide the network with the original image representation while preserving its dimensions.

The only requirement at this stage is consistency. Every image supplied to the network must have the same dimensions, including the height, width, and number of color channels.

Example: A model trained on RGB images with dimensions of $128\times 128\times 3$ expects every input image to follow that same structure

Understanding dimensions within a CNN can initially feel confusing because the shape of the data changes as it moves through different layers. However, this becomes much easier once we understand the mathematical operations behind each transformation. Later, we will introduce a simple formula that allows us to calculate these changes and track the dimensions throughout the entire network.

The Convolutional Layer

So, let's start with the most basic question:

What does the word 'convolution' actually mean, and why is it so important?

In the simplest mathematical definition, convolution is an operation that combines two functions by sliding one function over another and measuring the amount of overlap at each position.

Although this definition may sound abstract, the core idea is quite simple: convolution allows us to understand how two pieces of information interact as one moves across the other.

The important thing to remember is that convolution is not something that was invented specifically for neural networks. Long before CNNs existed, convolution was already being used extensively in fields such as mathematics, physics, engineering, and signal processing.

So convolution itself was not the invention that solved the problem of image recognition.

The brilliance of CNNs came from recognizing that this existing mathematical operation could be adapted into a learning mechanism. Instead of manually defining what features an image contains, we allow the network to learn which patterns are important by adjusting the parameters of its convolution filters during training.

This simple idea transformed convolution from a mathematical operation into one of the most powerful tools in modern computer vision.

Choosing How Deep We Go

At this point, we have two possible paths.

1. The Mathematical Path: The first option is to explore convolution from its original mathematical foundation. This involves diving into functions, continuous mathematics, integrals, and the calculus behind the operation. While this provides a complete understanding of convolution, it goes beyond what is necessary for understanding CNNs and is outside the scope of this article.

   For readers interested in the deeper mathematics, I will provide additional resources below where this topic is explored in much greater detail.

2. The CNN Path: The second option is to continue with a simplified interpretation of convolution as it is used within CNNs. For the purpose of understanding neural networks, we do not need to master the complete mathematical theory behind convolution. Instead, we will focus on the practical version used in computer vision.

In the context of Convolutional Neural Networks, convolution takes on a much simpler and more practical meaning

Instead of thinking about convolution as the mathematical operation between two continuous functions, we can think of it as a small matrix operation performed on an image.

A CNN uses a small square matrix of values called a filter or kernel. Initially, these values are assigned randomly.

The process of applying this filter to an image is what we refer to as convolving the image.

But what does that actually mean?

When a filter is placed over a small region of an image, each value in the filter is multiplied by the corresponding value in the image. All of these multiplied values are then added together to produce a single output value.

Mathematically, this can be represented as:

where:

• $x_i$ represents the pixel values from the input image
• $w_i$ represents the corresponding values within the filter
• $n$ represents the total number of values involved in the calculation

The result of this operation represents how strongly that particular region of the image matches the pattern encoded by the filter.

The filter then moves to the next position of the image and repeats the same calculation. By performing this operation across the entire image, the CNN creates a new representation called a feature map.

This feature map highlights the areas of the image where the filter detects the strongest patterns.

In practical implementations, a convolutional layer typically contains multiple filters, also known as kernels. Each filter contains a set of learnable parameters and is responsible for detecting different patterns within the input image.

For example, filters in earlier convolutional layers may learn to respond to simple visual features such as edges, lines, and textures. As information passes through deeper layers of the network, filters can learn increasingly complex patterns by combining simpler features from previous layers.

Initially, the values within these filters are assigned using a random initialization strategy. At this stage, the filters do not represent meaningful visual patterns. However, during training, the network evaluates its prediction error and adjusts these parameters to improve performance.

This process occurs through backpropagation, where the gradients of the error with respect to each filter parameter are calculated. An optimization algorithm then updates the filter values based on these gradients.

Through many iterations of training, the filters gradually learn to detect patterns that are useful for the specific task the CNN is designed to perform.

This entire convolution operation can be visualized as follows:

The Pooling Layer

After the convolutional layer extracts important features from an image, the next challenge is reducing the amount of information that the network needs to process.

This is where the pooling layer comes in.

The primary purpose of a pooling layer is to reduce the spatial dimensions of a feature map. It is important to note that pooling is applied to the feature maps produced by convolution, not the original input image.

By reducing the size of these feature maps, pooling decreases the number of computations required in later layers, making the network more efficient while retaining the most important information extracted by the convolutional layers.

So how does pooling achieve this?

Similar to convolution, pooling uses a small window that moves across the feature map. However, unlike a convolutional filter, this window does not contain learnable parameters. Instead, it applies a predefined operation to the values within each region.

In other words, the pooling window does not learn what to look for. Instead, it follows a fixed rule that determines how the information should be summarized.

There are several different types of pooling operations, each affecting the feature map in a different way:

• Max Pooling: Selects the maximum value within each region of the feature map. This is the most commonly used pooling technique because it preserves the strongest detected features.
• Min Pooling: Selects the minimum value within each region of the feature map. This is less commonly used but can be useful in specific applications.
• Average Pooling: Calculates the average value within each region, producing a smoother representation of the feature map.
• Global Max Pooling: Selects the single maximum value from the entire feature map, reducing the entire feature map to one value.

The choice of pooling operation affects how information is preserved and compressed as it moves through the network. In modern CNN architectures, max pooling and global average pooling are the most commonly encountered approaches.

The complete pooling operation can be visualized as follows:

The Fully Connected Layer

After the convolutional and pooling layers have extracted meaningful features from the input image, the resulting feature maps are passed to the fully connected layer. This stage of the network performs the final prediction based on the features learned during the earlier stages.

Depending on the problem, the fully connected layer can be used for either classification or regression tasks. In image classification, which is the focus of this discussion, its role is to assign the input image to one of a predefined set of classes.

Before being passed to the fully connected layer, the final set of feature maps is flattened into a one-dimensional vector. This vector serves as the input to the fully connected network, where each neuron is connected to every neuron in the subsequent layer. The network then combines the extracted features to produce the final prediction.

To better understand why CNNs delay this flattening step, it is useful to first consider the idea of spatial structure. In an image, neighboring pixels are often closely related and together form meaningful patterns such as edges, corners, textures, and shapes. These relationships between the positions of pixels are referred to as the image's spatial structure.

In a traditional fully connected neural network, the original image is flattened before being presented to the model. Although this process preserves every pixel value, it removes the explicit spatial arrangement of the image by converting it into a one-dimensional vector. As a result, the network must learn these spatial relationships from the data alone. In contrast, a CNN preserves the image's spatial structure throughout the convolutional and pooling stages, allowing it to learn meaningful visual features before the data is finally flattened for classification.

Key Characteristics of Convolutional Neural Networks

Convolutional Neural Networks are significantly more efficient than traditional fully connected neural networks when processing images. This efficiency arises from several key architectural properties that allow CNNs to exploit the spatial structure of image data.

Parameter Sharing

One of the defining characteristics of a CNN is parameter sharing.

Unlike a fully connected neural network, where every connection has its own unique weight, a convolutional layer applies the same set of parameters across different locations of the input image. These parameters are stored within a structure called a filter or kernel.

Before training begins, the values within each filter are initialized with small random numbers. At this stage, the filters do not represent any meaningful visual patterns. They are simply starting points from which the network begins learning.

A convolutional layer usually contains multiple filters, and each filter has its own independent set of learnable parameters. These filters do not share parameters with one another. Instead, each filter learns to detect different patterns that are useful for the task, such as edges, textures, shapes, or other visual features.

The key idea behind parameter sharing is that once a filter has been initialized, the same filter is reused across every spatial location of the input image. As the filter moves across the image, it applies the same weights at each position, producing a feature map that indicates where that particular pattern appears.

During training, the values inside each filter are continuously adjusted through backpropagation. Since the same filter is applied at multiple locations, the error contributions from all of these locations are combined to determine how the filter's parameters should change. An optimization algorithm then updates the weights to reduce the overall prediction error.

Through many training iterations, the initially random filters gradually learn meaningful patterns from the data. Because the same learned filter can detect its corresponding feature regardless of where it appears in the image, CNNs can recognize patterns efficiently while using significantly fewer parameters than fully connected networks.

Parameter sharing therefore provides two major advantages: it reduces the number of trainable parameters and allows the network to detect the same visual feature at different locations within an image.

Local Receptive Fields

Another important characteristic of CNNs is the use of local receptive fields.

Rather than examining an entire image at once, each filter only observes a small region of the input during a single convolution operation. This local region is known as the filter's receptive field.

By focusing on small regions, the network can efficiently learn simple visual patterns such as edges, corners, and textures. As multiple convolutional layers are stacked together, deeper layers gradually combine these simple patterns into more complex representations, including object parts and eventually entire objects.

Translation Equivariance

Because the same filter is applied across every location in an image, CNNs naturally exhibit a property known as translation equivariance.

This means that if a particular feature, such as an edge or a corner, appears in a different location within the image, the same filter can still detect it. In other words, the ability of the network to recognize a feature does not depend on its absolute position within the image.

This property makes CNNs particularly well suited to image analysis, where objects of interest may appear almost anywhere within the input.

Hierarchical Feature Learning

CNNs learn visual information in a hierarchical manner.

The filters in the earliest convolutional layers often learn simple patterns such as edges, lines, and textures. These low-level features are then combined by deeper layers to identify increasingly complex structures, including shapes, object parts, and ultimately complete objects.

This hierarchical learning process enables CNNs to automatically discover useful representations directly from raw image data, eliminating the need for manually engineered features that were common in earlier computer vision systems.

Controlling Feature Map Dimensions

In the previous sections, we saw that each convolutional layer transforms an input image or feature map into a new feature map. One question naturally follows: how large will this new feature map be?

The answer depends not only on the size of the input but also on several hyperparameters that control how the convolution operation is performed. By adjusting these hyperparameters, we can influence the dimensions of the output feature map, the amount of computation required, and, to some extent, the information that the network preserves as it learns.

The choice of these hyperparameters is largely problem-dependent. For some tasks, preserving fine details is essential, while for others, reducing computational cost may be more important. Consequently, selecting appropriate values is an important aspect of CNN design.

The three primary hyperparameters that determine the size of a feature map are:

• Stride: Stride specifies how many pixels the filter moves after each convolution operation. A stride of 1 moves the filter one pixel at a time, producing a larger feature map. Increasing the stride causes the filter to move further between calculations, reducing the spatial dimensions of the output while also lowering the computational cost.
• Padding: Padding involves adding extra pixels, most commonly zeros, around the border of the input image or feature map before convolution. Without padding, information near the edges of an image may be used less frequently, causing important boundary features to be lost. Padding helps preserve edge information and allows greater control over the output dimensions.
• Filter (Kernel) Size: The filter, or kernel, size determines the dimensions of the window used during convolution. Larger filters examine a wider region of the input at each step, increasing the receptive field but also requiring more computation. Smaller filters perform fewer calculations and are often stacked across multiple layers to learn complex patterns efficiently.

The size of the feature map can be predicted using a simple formula:

Conclusion

This breakdown introduced the fundamental concepts behind Convolutional Neural Networks, including convolution operations, feature extraction, pooling, fully connected layers, feature map transformations, and key architectural ideas such as local receptive fields, parameter sharing, and hierarchical feature learning.

However, these concepts represent only the foundation of CNNs. Many important ideas build upon this foundation, including activation functions, normalization techniques, optimization strategies, regularization methods, data augmentation, advanced CNN architectures, residual connections, attention mechanisms, and efficient convolution techniques.

Further topics to explore include dilated convolutions, which expand the receptive field of a network without significantly increasing the number of parameters, depth wise separable convolutions for improving computational efficiency, 3D CNNs for processing volumetric and temporal data, and specialized architectures for tasks beyond classification such as object detection and semantic segmentation.

Resources

Essential Video Lectures

If you only have time for one resource, I'd recommend this excellent overview of Convolutional Neural Networks:

• https://www.youtube.com/watch?v=pj9-rr1wDhM

MIT's Introduction to Deep Learning is one of my favorite lecture series on the subject. Lecture 3 provides a concise and intuitive introduction to Convolutional Neural Networks:

• https://www.youtube.com/watch?v=pqIcoskUuWs

If you're curious about the mathematics behind convolution, 3Blue1Brown has a fantastic visual explanation that makes the concepts much easier to understand:

• https://www.youtube.com/watch?v=KuXjwB4LzSA

Finally, the article itself is heavily inspired by the original survey paper that motivated me to write it:

• https://arxiv.org/abs/1511.08458

Blogs

If you'd like another perspective or want to explore the topic in more depth, here are a few resources I found particularly helpful.

Christopher Olah (co-founder of Anthropic) has one of the best technical blogs on the internet. His articles, including those on Convolutional Neural Networks, are beautifully written and full of intuitive explanations. It's the kind of writing I hope to reach one day.

• https://colah.github.io/

This overview from Viso.ai is a short, beginner-friendly introduction that provides a good overview of the history of Convolutional Neural Networks and where the field could be heading in the future.

• https://viso.ai/deep-learning/convolutional-neural-networks/

---

Note: This resource list will continue to grow as I discover more high-quality papers, lectures, and articles. Feel free to check back from time to time for updates.