Transfer Learning: Reusing What a Model Already Knows

Why Transfer Learning Works: Layered Learning

Deep neural networks learn hierarchically. Research (Yosinski et al., 2014) showed that:

• Layer 1 (early): learns low-level features (edges, corners, textures). These are highly general and transfer well across tasks.
• Layer 2-3 (middle): learns combinations of low-level features (shapes, object parts). Still mostly task-agnostic.
• Layer 4-5 (late): learns task-specific patterns (dog ears for dog classification, car wheels for vehicle classification). These are highly specific.

When you train a model on a large dataset like ImageNet (1.2M labeled images, 1000 classes, trained on millions of GPU hours), the early layers learn features that are useful for almost any vision task. If your task (say, identifying skin lesions in medical images) has only 10,000 labeled examples, training from scratch wastes data and time. Instead, reuse the pretrained early layers and adapt the later layers to your new task.

This is transfer learning: take a model trained on a source task, adapt it for a target task. You can do this in two ways:

1. Feature extraction: freeze all pretrained weights, replace the final layer, and train only the new head. Fast (minutes to hours) but potentially suboptimal.
2. Fine-tuning: unfreeze the later layers, lower the learning rate, and retrain the whole network on your data. Slower (hours to days) but often better final accuracy.

Feature extraction is faster and uses less memory. Fine-tuning is slower but can yield better results if you have enough data. The choice depends on how similar your target task is to ImageNet and how much data you have.

Real-world data efficiency

Yosinski et al. (2014) quantified the benefit. On Caltech-256 (30k images, 256 classes, smaller than ImageNet):

| Training Strategy | Accuracy on Caltech-256 | |---|---| | Train from scratch (random init) | ~70% | | Feature extraction (frozen ImageNet weights) | ~95% | | Fine-tune (lower learning rate) | ~97% |

The feature extraction model uses the same amount of data but achieves 25% higher accuracy just by reusing pretrained weights. This is the power of transfer learning.

How Much Data Do You Need?

A practical guideline based on dataset size and domain similarity (illustrative estimate):

| Target dataset size | Similar domain (e.g., ImageNet → ImageNet-variant) | Different domain (e.g., ImageNet → Medical Imaging) | |---|---|---| | < 10k examples | Freeze all but final layer (L1-4 frozen, only L5+fc trainable); LR = 0.001 | Unfreeze last block (L4-5 + fc trainable); LR = 0.0001 | | 10k - 100k examples | Unfreeze last 2-3 blocks; LR = 0.0001 (10x lower than scratch) | Unfreeze last half; LR = 0.00001 (100x lower) | | 100k - 1M examples | Unfreeze last half of network; LR = 0.00005 (5x lower) | Fine-tune entire network; LR = 0.00001 | | > 1M examples | Train from scratch or fine-tune all layers with LR = 0.0001 | Fine-tune all layers; LR = 0.00005 |

Domain similarity matters: If your target task is very different from ImageNet (e.g., medical imaging, satellite imagery, or X-ray scans), the pretrained features are less useful. You may need to unfreeze more layers earlier. Conversely, for similar tasks (dog breed classification, when source is dog/animal classification), shallow freezing works well.

Learning rate scheduling for fine-tuning

Use discriminative fine-tuning: assign different learning rates to different layer groups:

param_groups = [
    {'params': model.layer1.parameters(), 'lr': 0.00001},
    {'params': model.layer2.parameters(), 'lr': 0.00005},
    {'params': model.layer3.parameters(), 'lr': 0.0001},
    {'params': model.layer4.parameters(), 'lr': 0.0001},
    {'params': model.fc.parameters(), 'lr': 0.001},
]

optimizer = torch.optim.SGD(param_groups, momentum=0.9)

Earlier layers (closer to input) learn slowly, preserving general features. Later layers learn faster, adapting to new task. This is more stable than using a single learning rate.

Feature Extraction: Freeze and Replace

The simplest approach:

import torch
import torch.nn as nn
import torchvision.models as models

# Load a pretrained ResNet-18 (trained on ImageNet)
model = models.resnet18(pretrained=True)

# Inspect the model structure
print(model)
# It has a lot of layers, ending with:
# (fc): Linear(in_features=512, out_features=1000)  <- ImageNet has 1000 classes

# Freeze all parameters (gradient updates will be zero)
for param in model.parameters():
    param.requires_grad = False

# Replace the final layer for your task (e.g., 5 classes for your dataset)
num_classes = 5
model.fc = nn.Linear(in_features=512, out_features=num_classes)

# Only the new fc layer has requires_grad=True
print(f"Trainable parameters: {sum(p.numel() for p in model.parameters() if p.requires_grad)}")
# Should be ~512 * 5 + 5 = 2565 (tiny compared to the whole model's 11M parameters)

# Now train normally
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(
    filter(lambda p: p.requires_grad, model.parameters()),
    lr=0.001
)

# Dummy training loop
model.train()
for epoch in range(2):
    for batch_idx in range(5):
        x = torch.randn(32, 3, 224, 224)  # Batch of 32 RGB images
        y = torch.randint(0, num_classes, (32,))

        logits = model(x)
        loss = criterion(logits, y)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if batch_idx % 5 == 0:
            print(f"Epoch {epoch}, Batch {batch_idx}, Loss: {loss.item():.4f}")

print("Feature extraction training complete.")

This trains only the final layer (2565 parameters) while keeping the 10+ million pretrained features frozen. It is fast and often works well for similar tasks.

Fine-Tuning: Unfreeze Deeper Layers

For better results, gradually unfreeze layers:

# Start with the same pretrained model
model = models.resnet18(pretrained=True)

# Freeze everything initially
for param in model.parameters():
    param.requires_grad = False

# Replace the final layer
num_classes = 5
model.fc = nn.Linear(in_features=512, out_features=num_classes)

# Unfreeze the last residual block (layer4) and the fc layer
for param in model.layer4.parameters():
    param.requires_grad = True

for param in model.fc.parameters():
    param.requires_grad = True

# Create a optimizer with different learning rates for different layer groups
# (this is called "discriminative fine-tuning")
params_to_update = [
    {'params': model.layer4.parameters(), 'lr': 0.0001},
    {'params': model.fc.parameters(), 'lr': 0.001}
]

optimizer = torch.optim.SGD(params_to_update, momentum=0.9)

# Training loop (same as before, but now more weights are updated)
model.train()
for epoch in range(10):
    for batch_idx in range(20):
        x = torch.randn(32, 3, 224, 224)
        y = torch.randint(0, num_classes, (32,))

        logits = model(x)
        loss = nn.CrossEntropyLoss()(logits, y)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if batch_idx % 10 == 0:
            print(f"Epoch {epoch}, Batch {batch_idx}, Loss: {loss.item():.4f}")

print("Fine-tuning training complete.")

# Evaluate on a validation set (not shown here for brevity)
model.eval()
with torch.no_grad():
    val_logits = model(torch.randn(10, 3, 224, 224))
    print(f"Validation output shape: {val_logits.shape}")

Fine-tuning unfreezes layer4 (the last residual block) and trains it with a lower learning rate (0.0001) than the new fc layer (0.001). This prevents the pretrained features from changing too drastically.

A Complete Transfer Learning Pipeline

Here is a realistic example using a real dataset pattern:

import torch
import torch.nn as nn
import torchvision.models as models
import torchvision.transforms as transforms
from torch.utils.data import DataLoader, TensorDataset

# 1. Define preprocessing (must match the ImageNet preprocessing)
preprocess = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(
        mean=[0.485, 0.456, 0.406],
        std=[0.229, 0.224, 0.225]
    )
])

# 2. Load pretrained model
model = models.resnet18(pretrained=True)

# 3. Freeze earlier layers, unfreeze later layers
for name, param in model.named_parameters():
    if 'layer3' in name or 'layer4' in name or 'fc' in name:
        param.requires_grad = True
    else:
        param.requires_grad = False

# 4. Replace the classification head
num_classes = 10
model.fc = nn.Linear(512, num_classes)

# 5. Set up training
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = model.to(device)

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(
    filter(lambda p: p.requires_grad, model.parameters()),
    lr=0.001
)

# 6. Training loop
def train_epoch(model, loader, criterion, optimizer, device):
    model.train()
    total_loss = 0
    for batch_x, batch_y in loader:
        batch_x, batch_y = batch_x.to(device), batch_y.to(device)

        logits = model(batch_x)
        loss = criterion(logits, batch_y)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        total_loss += loss.item()

    return total_loss / len(loader)

# 7. Evaluation loop
def evaluate(model, loader, device):
    model.eval()
    correct, total = 0, 0
    with torch.no_grad():
        for batch_x, batch_y in loader:
            batch_x, batch_y = batch_x.to(device), batch_y.to(device)
            logits = model(batch_x)
            preds = logits.argmax(dim=1)
            correct += (preds == batch_y).sum().item()
            total += batch_y.size(0)

    return correct / total

# Create dummy dataloaders (in practice, use real data)
train_x = torch.randn(1000, 3, 224, 224)
train_y = torch.randint(0, num_classes, (1000,))
train_dataset = TensorDataset(train_x, train_y)
train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True)

val_x = torch.randn(200, 3, 224, 224)
val_y = torch.randint(0, num_classes, (200,))
val_dataset = TensorDataset(val_x, val_y)
val_loader = DataLoader(val_dataset, batch_size=32)

# Train for a few epochs
for epoch in range(3):
    train_loss = train_epoch(model, train_loader, criterion, optimizer, device)
    val_acc = evaluate(model, val_loader, device)
    print(f"Epoch {epoch}: Train Loss: {train_loss:.4f}, Val Acc: {val_acc:.3f}")

This pipeline:

1. Loads a pretrained ResNet-18.
2. Freezes early layers, unfreezes layer3, layer4, and fc.
3. Replaces the final layer.
4. Trains with a lower learning rate.
5. Evaluates on a validation set.

Feature Extraction vs. Fine-Tuning Trade-offs

Here is a comprehensive comparison of when to use each approach:

| Aspect | Feature Extraction (Freeze All) | Fine-Tuning (Unfreeze Layers) | Training from Scratch | |---|---|---|---| | Training time | Very fast (~hours) | Moderate (~1-3 days) | Very slow (~7-14 days) | | Data required | ~1k-10k examples | ~10k-100k examples | 100k+ examples or full domain-matched dataset | | Compute (GPU hours) | 10-50 | 100-500 | 1000+ | | Final accuracy | Good (85-95%) if domain similar | Excellent (90-98%) if sufficient data | Excellent (92-98%) with large data | | Risk of overfitting | Low: most weights frozen | Moderate: learnable params can overfit | High: many degrees of freedom | | When it succeeds | ImageNet → similar visual task | ImageNet → somewhat different domain | Medical imaging, satellite, very specialized tasks | | When it fails | Domain very different (medical imaging) | Not enough data to fine-tune well | Already have pretrained model (wastes compute) |

Practical example: Detecting defects in manufacturing images (10k labeled images):

• Feature extraction: ResNet-18 on ImageNet features; accuracy ~92%, trained in 2 hours
• Fine-tuning (last 2 blocks): ResNet-18 with last 2 blocks unfrozen; accuracy ~95%, trained in 8 hours
• From scratch: Random init ResNet-18; accuracy ~80-85%, trained in 40 hours (and overfits due to limited data)

Fine-tuning is the sweet spot here: better than feature extraction without the compute cost of training from scratch.

Domain Adaptation: When Pretrained Features Transfer Less

Sometimes pretrained ImageNet features transfer poorly. Common scenarios:

| Domain | ImageNet Transfer Effectiveness | Recommendation | |---|---|---| | Natural images (ImageNet-like) | Excellent (85-95% transfer) | Feature extraction or light fine-tuning | | Medical imaging (X-ray, CT, MRI) | Moderate (50-70% transfer) | Unfreeze more layers; consider domain-specific pretrained models | | Satellite/Aerial imagery | Moderate-good (60-80% transfer) | Fine-tune last 3-4 blocks | | Microscopy | Moderate (40-70% transfer) | Heavy fine-tuning or consider specialized pretrained models | | Thermal/Infrared | Poor (20-40% transfer) | Train from scratch or use domain-specific pretraining | | Synthetic data (computer-rendered) | Poor (30-50% transfer) | Domain adaptation techniques needed |

For medical imaging specifically: Yosinski et al. (2014) showed that features from ImageNet transfer reasonably (85% accuracy on Caltech-256), but specialized models (trained on similar medical datasets) transfer better. Consider using pretrained medical models from:

• ResNet trained on ChexPert (chest X-rays): better for other X-ray tasks
• ResNet trained on NIH dataset (general medical): better for varied medical imaging

When NOT to Use Transfer Learning

Transfer learning is powerful but not always the right choice:

1. Very different domain: If your task is drastically different from ImageNet (e.g., medical imaging, audio spectrograms), the pretrained features might not transfer well. Experiment: try feature extraction first; if it underperforms, fine-tune or train from scratch.

2. Very large target dataset: If you have 10M labeled examples, training from scratch often outperforms transfer learning. The cost of pretraining is amortized over massive compute.

3. No relevant pretrained model exists: If there is no pretrained model for your domain, you have to train from scratch or find synthetic pretraining data.

4. Extreme parameter mismatches: If the pretrained model and your target task have very different architectures (e.g., pretrained on images with a CNN, but you need an RNN for sequences), transfer learning requires more care and may not help.

Case Study: Real-world Transfer Learning

Task: Classify dog breeds from photos. Dataset: 5,000 images of 20 dog breeds (250 images per class).

Scenario 1: Feature Extraction

model = models.resnet50(pretrained=True)
for param in model.parameters():
    param.requires_grad = False
model.fc = nn.Linear(2048, 20)  # 20 dog breeds

# Train only final layer: 2000 GPU seconds (~30 min)
# Final accuracy: 93%

Scenario 2: Fine-tuning last block

model = models.resnet50(pretrained=True)
for param in model.layer1.parameters():
    param.requires_grad = False
for param in model.layer2.parameters():
    param.requires_grad = False
for param in model.layer3.parameters():
    param.requires_grad = False
# layer4 and fc trainable

# Train with LR=0.0001: 10000 GPU seconds (~3 hours)
# Final accuracy: 95%

Scenario 3: Progressive fine-tuning

• Stage 1 (100 epochs): Train only fc; accuracy reaches 92%
• Stage 2 (50 epochs): Unfreeze layer4; train at LR=0.00001; accuracy reaches 96%
• Stage 3 (30 epochs): Unfreeze layer3; train at LR=0.000005; accuracy reaches 96.5%

Total: 15000 GPU seconds (~4 hours), final accuracy 96.5%

Result: Feature extraction is 8× faster but reaches 93% accuracy. Progressive fine-tuning takes 4 hours but reaches 96.5% — a 3.5% accuracy boost, which is significant for production.

Advanced: Progressive Fine-Tuning

For large datasets, unfreeze layers progressively:

def progressive_fine_tune(model, loaders, num_epochs_per_stage):
    """
    Unfreeze layers one by one and fine-tune.

    Assumes model structure: layer1, layer2, layer3, layer4, fc
    """
    layer_names = ['layer1', 'layer2', 'layer3', 'layer4']

    for stage, layer_name in enumerate(layer_names):
        print(f"Stage {stage}: Unfreezing {layer_name}")

        # Unfreeze this layer
        for param in getattr(model, layer_name).parameters():
            param.requires_grad = True

        # Adjust learning rate (lower for earlier layers)
        lr = 0.001 / (2 ** (len(layer_names) - stage - 1))

        optimizer = torch.optim.Adam(
            filter(lambda p: p.requires_grad, model.parameters()),
            lr=lr
        )

        # Train for a few epochs
        for epoch in range(num_epochs_per_stage):
            train_loss = train_epoch(
                model, loaders['train'], criterion, optimizer, device
            )
            val_acc = evaluate(model, loaders['val'], device)
            print(f"  Epoch {epoch}: Loss {train_loss:.4f}, Val Acc {val_acc:.3f}")

This unfreezes layer1, trains, then unfreezes layer2, trains, etc. Each stage uses a lower learning rate for older layers, stabilizing the training process.

Common mistake: using the wrong preprocessing

Pretrained models like ResNet were trained on ImageNet images that were resized to 224x224, normalized with specific mean/std (the ImageNet statistics). If you apply different preprocessing, the model will perform poorly even though it is pretrained.

Always use the same preprocessing as the original training:

# Correct: use ImageNet normalization
transforms.Normalize(
    mean=[0.485, 0.456, 0.406],
    std=[0.229, 0.224, 0.225]
)

# Incorrect: using different normalization
transforms.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5])
# This will drastically hurt performance

Another mistake: forgetting to call model.eval() during evaluation. In training mode, dropout and batch norm behave differently, leading to inflated evaluation metrics. Always switch to eval mode and disable gradient tracking with torch.no_grad().