From a Single Neuron to a Layer

A Single Neuron: Input to Output

An artificial neuron is remarkably simple. It takes a vector of inputs, multiplies each by a weight, sums the results, adds a bias, and passes the sum through an activation function. This design is inspired by biological neurons, where inputs are dendrites, weights are synaptic strengths, and the activation function models the neuron's firing threshold.

The diagram on the left shows a single neuron with three inputs. Each input x_i has an associated weight w_i. The weighted inputs are summed, a bias b is added, and the result is passed through an activation function (e.g., ReLU) to produce the output.

Mathematically, the neuron computes:

z = w₁*x₁ + w₂*x₂ + w₃*x₃ + b     (pre-activation, or "logit")
output = σ(z)                      (post-activation)

Or in vector form:

z = w · x + b = Σᵢ(wᵢ * xᵢ) + b
output = σ(z)

where · denotes the dot product and σ is an activation function (ReLU, sigmoid, tanh, etc.).

Concrete example: Suppose x = [2, 1, -1], w = [0.5, -0.3, 0.8], b = 0.1, and σ = ReLU. Then:

• z = 0.5(2) + (-0.3)(1) + 0.8(-1) + 0.1 = 1.0 - 0.3 - 0.8 + 0.1 = 0.0
• output = ReLU(0.0) = 0.0

Why Three Components: Weights, Bias, and Activation?

Weights (w) control how much each input contributes to the neuron's decision. A large weight means that input has a strong influence; a weight near zero means it is nearly ignored.

Bias (b) shifts the neuron's threshold. Without bias, the neuron cannot move its decision boundary away from the origin. Bias allows the neuron to "fire" even when all inputs are zero.

Activation function introduces non-linearity. Without it, stacking neurons would be equivalent to a single linear transformation (matrix multiplication), and the network could only learn linear patterns. The activation function enables learning non-linear relationships.

Common activation functions include:

• ReLU (Rectified Linear Unit): max(0, z) — zeros out negative values, enables training very deep networks.
• Sigmoid: 1 / (1 + exp(-z)) — outputs a probability between 0 and 1, useful for output layers in binary classification.
• Tanh: (exp(z) - exp(-z)) / (exp(z) + exp(-z)) — outputs between -1 and 1, centered, useful in RNNs.

Why activation matters: Without an activation function, the neuron outputs z = w·x + b, which is just a linear transformation. Stacking linear transformations (neuron → neuron → ... → neuron) still yields a linear transformation overall (matrix multiplication is associative). Linear models can only learn linear patterns. The activation function σ(z) introduces non-linearity, allowing neurons to learn curves, boundaries, and complex relationships.

| Component | Role | Impact on Learning | |-----------|------|------| | Weight w_i | Scales input's influence | Large w → input dominates; small w → input ignored | | Bias b | Shifts decision threshold | Without bias, threshold locked at origin | | Activation σ | Non-linearity | Without σ, network is just linear; network can't learn non-linear patterns |

Why Bias Matters: A Visual Example

Consider a binary classification problem: separate points into two regions based on their position. Without bias, the decision boundary must pass through the origin. With bias, it can shift to fit the data better.

import matplotlib.pyplot as plt

# Data with offset from origin
X_class0 = np.random.randn(30, 2) + np.array([2, 2])
X_class1 = np.random.randn(30, 2) + np.array([-2, -2])

# Decision boundary without bias: passes through origin
w_no_bias = np.array([1, 1])  # Simple weights
boundary_x = np.linspace(-5, 5, 100)
boundary_y_no_bias = -w_no_bias[0] / w_no_bias[1] * boundary_x  # Passes through origin

# Decision boundary with bias: can shift
w_with_bias = np.array([1, 1])
b_with_bias = 2.0
boundary_y_with_bias = -(w_with_bias[0] * boundary_x + b_with_bias) / w_with_bias[1]

fig, axes = plt.subplots(1, 2, figsize=(12, 5))

for ax, boundary_y, title, uses_bias in zip(
    axes,
    [boundary_y_no_bias, boundary_y_with_bias],
    ["Without Bias", "With Bias"],
    [False, True]
):
    ax.scatter(X_class0[:, 0], X_class0[:, 1], label='Class 0', alpha=0.6, color='blue')
    ax.scatter(X_class1[:, 0], X_class1[:, 1], label='Class 1', alpha=0.6, color='red')
    ax.plot(boundary_x, boundary_y, 'k--', linewidth=2, label='Decision boundary')
    ax.set_xlim([-5, 5])
    ax.set_ylim([-5, 5])
    ax.axhline(y=0, color='gray', linestyle='-', alpha=0.3)
    ax.axvline(x=0, color='gray', linestyle='-', alpha=0.3)
    ax.set_title(f"{title} (Bias = {b_with_bias if uses_bias else 0})")
    ax.legend()
    ax.set_aspect('equal')

plt.tight_layout()
plt.show()

The diagram shows: without bias, the boundary is locked at the origin and misclassifies many points. With bias, it shifts to better separate the classes. Bias is not optional—it is fundamental.

From Scratch: A Single Neuron in NumPy

Let's implement a single neuron that processes a single input sample:

import numpy as np

# Define a simple activation function: ReLU
def relu(z):
    return np.maximum(0, z)

# Single neuron with 3 inputs
weights = np.array([0.5, -0.3, 0.8])  # w₁, w₂, w₃
bias = 0.1  # b

# Sample input
x = np.array([2.0, 1.0, -1.0])  # x₁, x₂, x₃

# Forward pass
z = np.dot(weights, x) + bias  # Weighted sum + bias
output = relu(z)  # Apply activation

print(f"Input: {x}")
print(f"Weighted sum (z): {z:.4f}")
print(f"Output (after ReLU): {output:.4f}")

Step-by-step:

1. Weighted sum: z = 0.5*2.0 + (-0.3)1.0 + 0.8(-1.0) + 0.1 = 1.0 - 0.3 - 0.8 + 0.1 = 0.0
2. Activation: output = ReLU(0.0) = 0.0

Change one input slightly and re-run; you will see how weights control the neuron's sensitivity to each input.

From One Neuron to a Layer: Many Neurons in Parallel

A fully connected layer stacks many neurons, each with its own weights and bias. A layer with 50 neurons means 50 independent computations happening in parallel, each with its own weight vector.

For a layer with m inputs and n neurons:

• Each neuron has m weights → total m*n weights.
• Each neuron has 1 bias → total n biases.
• The output is n values.

Why neurons don't interfere: Each neuron's output is independent of all others. You can compute all n neuron outputs in a single matrix operation, taking advantage of hardware parallelism (GPUs excel at this). This is why deep learning on modern GPUs is so fast.

In matrix form:

Z = X @ W + b
A = activation(Z)

where:

• X is a batch of inputs with shape (batch_size, m).
• W is the weight matrix with shape (m, n).
• b is the bias vector with shape (n,).
• Z is the pre-activation output with shape (batch_size, n).
• A is the post-activation output with shape (batch_size, n).

The @ operator is matrix multiplication.

Concrete example: If X is (32, 10)—32 samples, 10 features—and W is (10, 64), then Z = X @ W produces shape (32, 64), giving 32 samples with 64 neuron outputs each. Adding b (shape 64) via broadcasting ensures each sample's 64 neurons get the bias term.

Implementing a Layer from Scratch

import numpy as np

class FullyConnectedLayer:
    """A simple fully connected layer."""
    def __init__(self, input_size, output_size):
        # Initialize weights randomly (small values)
        self.weights = np.random.randn(input_size, output_size) * 0.01
        # Initialize biases to zero
        self.bias = np.zeros((1, output_size))

    def forward(self, X):
        """Forward pass.

        Args:
            X: Input batch with shape (batch_size, input_size)

        Returns:
            A: Output batch with shape (batch_size, output_size)
        """
        Z = X @ self.weights + self.bias  # Linear transformation
        A = np.maximum(0, Z)  # ReLU activation
        return A

# Create a layer: 10 inputs, 5 outputs
layer = FullyConnectedLayer(input_size=10, output_size=5)

# Sample batch of 3 samples, 10 features each
X_batch = np.random.randn(3, 10)

# Forward pass
output = layer.forward(X_batch)
print(f"Input shape: {X_batch.shape}")
print(f"Weight shape: {layer.weights.shape}")
print(f"Output shape: {output.shape}")
print(f"Output:\n{output}")

Key insight: Each row of the output corresponds to one sample in the batch, and each column corresponds to one neuron's output. The entire computation happens in parallel via matrix multiplication—this is why neural networks are efficient on GPUs, which excel at matrix operations.

Connecting to PyTorch: nn.Linear

PyTorch's nn.Linear module does exactly what we implemented above, but with highly optimized code and gradient computation built-in:

import torch
import torch.nn as nn

# Create a fully connected layer: 10 inputs, 5 outputs
layer = nn.Linear(in_features=10, out_features=5)

# Create a batch of 3 samples, 10 features each
X_batch = torch.randn(3, 10)

# Forward pass (note: no activation function here; nn.Linear only does the linear part)
Z = layer(X_batch)
print(f"Input shape: {X_batch.shape}")
print(f"Weight shape: {layer.weight.shape}")  # Note: weight shape is (out_features, in_features)
print(f"Output shape (before activation): {Z.shape}")

# Apply activation manually
output = torch.relu(Z)
print(f"Output shape (after ReLU): {output.shape}")

Notice that PyTorch's weight matrix has shape (out_features, in_features) = (5, 10), while our NumPy version used (in_features, out_features) = (10, 5). Both work, but PyTorch transposes the weight internally for efficiency. The bias still has shape (out_features,) = (5,).

Stacking Layers: Building a Multi-Layer Network

A single layer is not powerful enough for complex tasks. We stack multiple layers, with the output of one layer becoming the input to the next:

# Multi-layer network with PyTorch
class SimpleNetwork(nn.Module):
    def __init__(self):
        super(SimpleNetwork, self).__init__()
        self.fc1 = nn.Linear(10, 64)  # Input: 10, Hidden: 64
        self.fc2 = nn.Linear(64, 32)  # Hidden: 64, Output: 32
        self.fc3 = nn.Linear(32, 2)   # Final output: 2 classes

    def forward(self, x):
        x = torch.relu(self.fc1(x))  # Layer 1 + ReLU
        x = torch.relu(self.fc2(x))  # Layer 2 + ReLU
        x = self.fc3(x)              # Output layer (no activation for logits)
        return x

# Create network and process a batch
network = SimpleNetwork()
X_batch = torch.randn(5, 10)  # 5 samples, 10 features
output = network(X_batch)
print(f"Input shape: {X_batch.shape}")
print(f"Output shape: {output.shape}")  # (5, 2) — predictions for 5 samples, 2 classes

Why stack layers?

• Each layer learns abstract features from the previous layer.
• Layer 1 might learn simple patterns (edges, colors in images).
• Layer 2 might combine those patterns into higher-level features (shapes, textures).
• Layer 3 might recognize objects by combining shapes and textures.
• This hierarchy of abstraction is what makes deep learning powerful.

A Complete Example: Training a Two-Layer Network

Here is a minimal example of training a two-layer network on synthetic data:

import torch
import torch.nn as nn
import torch.optim as optim

# Create synthetic classification data
X_train = torch.randn(100, 5)
y_train = (X_train[:, 0] + X_train[:, 1] > 0).long()  # Simple binary rule

# Define a two-layer network
model = nn.Sequential(
    nn.Linear(5, 16),
    nn.ReLU(),
    nn.Linear(16, 2)
)

# Loss and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.01)

# Training loop
for epoch in range(50):
    # Forward pass
    logits = model(X_train)
    loss = criterion(logits, y_train)

    # Backward pass
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    if (epoch + 1) % 10 == 0:
        predictions = logits.argmax(dim=1)
        accuracy = (predictions == y_train).float().mean()
        print(f"Epoch {epoch+1}: Loss={loss.item():.4f}, Accuracy={accuracy.item():.2%}")

After training, the network learns to classify the synthetic data. The first layer transforms the 5 inputs into 16 intermediate features; the second layer maps those 16 features to 2 class logits.

Weight Initialization: Starting in the Right Place

How you initialize weights affects training speed and final performance. Random initialization is necessary—identical weights mean all neurons learn the same thing—but the scale matters.

Naive initialization (all ones or all zeros):

# WRONG: All weights are identical
model[0].weight.data.fill_(1.0)
# All neurons will have identical outputs; training fails

Standard random initialization:

# Decent starting point: random from [-0.1, 0.1]
nn.init.uniform_(model[0].weight, -0.1, 0.1)
nn.init.uniform_(model[0].bias, -0.1, 0.1)

Xavier/Glorot initialization (automatically used by PyTorch's default):

# Named after Xavier Glorot; scales variance by layer size
# Good for sigmoid/tanh activations
nn.init.xavier_uniform_(model[0].weight)

He initialization (for ReLU):

# Accounts for ReLU zeroing out half the activations
# Better for ReLU than Xavier
nn.init.kaiming_uniform_(model[0].weight, nonlinearity='relu')

PyTorch uses He initialization by default for most layers, which works well with ReLU. If you use tanh or sigmoid, Xavier might be better.

Weight initialization comparison:

| Method | Formula | Variance | Best For | Training Speed | Risk if Mismatched | |--------|---------|----------|----------|-----|----| | Uniform [-0.1, 0.1] | Manual bounds | Low (unstable) | Quick baselines | Medium | Dead neurons, saturation | | Xavier/Glorot | √(6/(n_in + n_out)) | Layer-size aware | Sigmoid/Tanh | Fast | Exploding/vanishing gradients with ReLU | | He | √(2/n_in) | Accounts for ReLU sparsity | ReLU, Leaky ReLU | Fast | Saturation for sigmoid/tanh | | Orthogonal | QR decomposition | Preserves gradient norm | Very deep networks | Very fast | Overkill for shallow networks | | Normal(0, 0.01) | Small Gaussian | Conservative | Safe baseline | Slow | Underflow, weak initial signal |

Understanding Neuron Saturation

A crucial but often-overlooked phenomenon: if weights are too large, neurons saturate—their outputs become extreme (all 0 or all 1 for sigmoid), and gradients vanish. Careful initialization prevents this.

# Demonstrate saturation
import matplotlib.pyplot as plt

x = np.linspace(-10, 10, 100)
sigmoid = 1 / (1 + np.exp(-x))
sigmoid_grad = sigmoid * (1 - sigmoid)

plt.figure(figsize=(10, 5))
plt.subplot(1, 2, 1)
plt.plot(x, sigmoid, 'b-', linewidth=2)
plt.axhline(y=0, color='k', linestyle='-', alpha=0.3)
plt.axhline(y=1, color='k', linestyle='-', alpha=0.3)
plt.axvline(x=-5, color='r', linestyle='--', alpha=0.5, label='Saturation region')
plt.axvline(x=5, color='r', linestyle='--', alpha=0.5)
plt.xlabel('Input (z)')
plt.ylabel('sigmoid(z)')
plt.title('Sigmoid Saturation: Gradient → 0 at extremes')
plt.legend()

plt.subplot(1, 2, 2)
plt.plot(x, sigmoid_grad, 'g-', linewidth=2)
plt.fill_between(x, sigmoid_grad, where=(x < -5) | (x > 5), alpha=0.3, label='Saturated region')
plt.xlabel('Input (z)')
plt.ylabel("sigmoid'(z)")
plt.title('Gradient of Sigmoid: Near-Zero in Saturation')
plt.legend()
plt.tight_layout()
plt.show()

When inputs fall outside [-5, 5], the sigmoid gradient is negligible. Initialization that produces large inputs leads to immediate saturation and vanishing gradients. He initialization keeps inputs in a reasonable range.

From Fully Connected to Beyond

Fully connected layers work well for tabular data but are inefficient for images (too many parameters) and miss spatial structure. This motivates:

• Convolutional layers (for images): Share weights across spatial positions, dramatically reducing parameters.
• Recurrent layers (for sequences): Process input step-by-step, with hidden state linking time steps.
• Attention mechanisms (for sequences): Allow any position to attend to any other, without sequential processing.

All build on the foundation of neurons and layers. Understanding the linear transformation + activation pattern is the first step to understanding any architecture.

Common Mistake: Forgetting the Activation Function

A frequent beginner error is stacking linear layers without activation functions:

# WRONG: Stacking linear layers without activation
model = nn.Sequential(
    nn.Linear(10, 64),
    # Missing activation!
    nn.Linear(64, 32),
    # Missing activation!
    nn.Linear(32, 2)
)

This network is mathematically equivalent to a single nn.Linear(10, 2) because composing linear functions yields another linear function. The network cannot learn non-linear patterns.

Always add activation functions between layers:

# RIGHT: Linear layers with activation functions
model = nn.Sequential(
    nn.Linear(10, 64),
    nn.ReLU(),  # Activation after first layer
    nn.Linear(64, 32),
    nn.ReLU(),  # Activation after second layer
    nn.Linear(32, 2)  # No activation after output (for logits)
)

Do not add an activation function after the final output layer (unless you have a specific reason, such as a sigmoid for binary classification with a different loss function). The loss function (e.g., CrossEntropyLoss) expects raw logits.