Introduction to Neural Networks and Deep Learning

Brief history

Early neural networks (1943–1997)

The deep learning revolution (2012–present)

Enabling factors:

• GPU computing power (NVIDIA CUDA)
• Big Data availability (ImageNet, Common Crawl)
• Algorithmic advances (Dropout, BatchNorm, Adam)
• Massive investment from tech industry

Every major breakthrough above relied on neural networks. The rest of this notebook explains what they are and how they learn.

What is a neural network?

A neural network is a function that learns from examples. At its core is a simple building block: the perceptron (artificial neuron).

3Blue1Brown — But what is a neural network?

The perceptron

A perceptron takes numbers in, computes a weighted sum, and applies a simple rule:

$\hat{y} = f\!\left(\sum_{i=1}^{n} w_i x_i + b\right)$

Think of it as a weighted vote:

1. Inputs ($x_i$): your features (e.g., pixel intensities, sensor readings)
2. Weights ($w_i$): how much the model "cares" about each feature
3. Bias ($b$): the baseline output when all inputs are zero
4. Activation ($f$): a non-linear function that shapes the output

Key insight: without an activation function (i.e., $f(x) = x$), a single perceptron is exactly equivalent to linear regression. (Later we will see how adding specific activations turns it into models like logistic regression).

Linear separability and the XOR problem

A perceptron draws a single straight decision boundary. If your classes can be separated by one line (in 2D) or hyperplane (in higher dimensions), the perceptron works. If not — it fails.

Why does XOR matter? It's the simplest example of a problem where you need combinations of conditions ("A or B, but not both"). In real data, this appears as feature interactions — fraud detection ("high amount AND new device"), medical diagnosis, etc. A single boundary can't capture these patterns.

Deep dive: see complementary/xor_linearity.ipynb for a full interactive exploration with decision boundary visualizations.

Multilayer perceptrons (MLPs)

To solve problems like XOR, we stack perceptrons into layers:

Each hidden layer learns intermediate representations — features that a single perceptron can't capture:

Critical detail: stacking linear layers without activations between them collapses back to a single linear transformation. You need repeated blocks of:

1. Linear layer → learns a weighted combination
2. Activation function → bends the output non-linearly

This is what makes neural networks more powerful than linear models.

Activation functions

Activation functions are the key to a neural network's power. Without them, no matter how many layers you stack, the entire network mathematically collapses into a single linear transformation (a "big linear model"), making it impossible to solve complex, non-linear problems like XOR. By introducing non-linearity after each linear layer, activation functions allow the network to bend and fold the feature space to learn highly complex patterns.

Rules of thumb:

• Use ReLU in hidden layers unless you have a reason not to
• Use sigmoid for binary outputs, softmax for multi-class outputs
• The output activation is part of your model's "API contract" — it shapes what the numbers mean

The following plot lets you compare these functions side by side. Notice how ReLU is the simplest (just zero out negatives), while sigmoid and tanh "squeeze" all values into a bounded range:

Training overview

Training a neural network means adjusting the weights so predictions match the targets.

3Blue1Brown — Gradient descent, how neural networks learn

Loss: "how wrong is the model?"

A loss function assigns a number to "how bad the predictions are." Lower is better.

Gradient descent: "walk downhill"

Gradient descent is an optimization algorithm: it iteratively adjusts parameters in the direction that reduces the loss the most.

The learning rate controls step size: too large → unstable, too small → slow.

Deep dive: see complementary/gradient_descent_from_scratch.ipynb for a from-scratch numpy implementation with 3D cost landscape visualizations.

Backpropagation: "pass the blame back"

3Blue1Brown — Backpropagation

Backpropagation computes "which weights should change, and by how much" to reduce the loss.

You don't need to do derivatives by hand. The mental model:

1. Forward pass builds a computation graph (chain of tensor operations)
2. loss.backward() walks the graph backwards, computing gradients for every parameter
3. Optimizer (SGD, Adam, …) uses those gradients to update parameters

This is what PyTorch's autograd does automatically.

The core training loop

In PyTorch, training always follows this pattern:

for epoch in range(num_epochs):
    # 1. Forward pass
    predictions = model(inputs)
    
    # 2. Compute loss
    loss = loss_fn(predictions, targets)
    
    # 3. Zero gradients (reset from previous step)
    optimizer.zero_grad()
    
    # 4. Backward pass (compute gradients)
    loss.backward()
    
    # 5. Update parameters
    optimizer.step()

Two important modes:

• model.train() — enables dropout, batch norm updates (use during training)
• model.eval() + torch.no_grad() — disables gradients (use during inference, faster)

Deep learning architectures (preview)

The MLP we've described is just one type of neural network. Modern deep learning uses specialized architectures for different data types:

Each will be covered in detail in later notebooks. This course focuses on the fundamentals that all of them share: neurons, layers, activations, loss functions, gradient descent, and backpropagation.

Resources (highly recommended)

Video series:

• 3Blue1Brown — Neural Networks — visual, intuitive explanations (English)
• DotCSV — Neural Networks — excellent series (Spanish)
• ChatGPT is made from 100 million of these [The Perceptron] — (English)

Interactive:

• TensorFlow Playground — experiment with neural networks in your browser following the provided guide and study the excelent article "Understanding neural networks with TensorFlow Playground".