RNN vs Transformer

Evaluation

DistilBERT Performance

After training, the model was evaluated on the held-out test set (7,112 samples) using multiple metrics to provide a comprehensive view of classification performance.

Image 4: DistilBERT Confusion Matrix

Image 5: DistilBERT F1 Scores Bar Chart

RNN Performance

BiLSTM with Attention

The BiLSTM model was evaluated after 20 epochs, reaching its peak validation accuracy of 69.72% at epoch 18.

Image 6: BiLSTM with Attention Confusion Matrix

Image 7: BiLSTM with Attention F1 Scores Bar Chart

Discussion

Key Insights and Discussion

Model Behavior and Overfitting

• The gap between validation accuracy and test accuracy (e.g., DistilBERT 78.89% vs 73.88%) suggests mild overfitting. This is partly attributed to the relatively small training set (~8,500 samples) distributed across 20 diverse classes.
• Class weighting in the loss function helped improve recall for underrepresented categories like talk.religion.misc and talk.politics.misc, though they still remain the most challenging for both architectures.

Semantic Analysis

• Classes with clear topical boundaries (e.g., rec.sport.hockey, sci.space) consistently achieved higher F1 scores (~0.80+ for DistilBERT, ~0.70+ for BiLSTM) due to unique technical vocabularies.
• Semantically overlapping classes (e.g., politics vs guns, or different computer hardware categories) showed higher confusion rates, which is expected given their shared terminology.

Error Patterns (Confusion Matrix Analysis)

Specific misclassifications reveal systemic issues in text understanding:

• Religion: talk.religion.misc is frequently confused with alt.atheism and soc.religion.christian due to high lexical overlap in philosophical discussions.
• Politics: talk.politics.misc is often misidentified as guns or mideast subcategories, as general political discussions often touch on these specific themes.
• Technical brevity: Short documents in technical categories like comp.graphics are sometimes confused with mac.hardware when they lack specific keywords but share general technical terms.

Comparison

Systematic Comparison

A systematic comparison will be conducted across three dimensions:

1. Architecture comparison: DistilBERT (67M params) vs LSTM — comparing transformer-based models against a recurrent baseline to evaluate the impact of self-attention and pretrained representations on text classification performance.
2. Fine-tuning strategy comparison: For each transformer model, three strategies are evaluated:
   • Freeze backbone (train classification head only)
   • Full fine-tuning (train all layers from start with layerwise LR)
   • Hybrid (train head first, then unfreeze backbone with differential LR)
3. Model efficiency comparison: Accuracy vs F1 Macro, Training time, Inference latency and Number of Parameters — to determine whether the larger BERT-base model justifies its additional computational cost over DistilBERT.

Results will be reported in the following format after all experiments are completed:

Key Findings: Transformer vs RNN

The ~11% accuracy gap between BiLSTM (62.9%) and BERT (73.88%) demonstrates that:

• Contextualized representations are essential: Static GloVe embeddings cannot capture the polysemy and context-dependent meanings that DistilBERT/BERT handle via self-attention.
• Attention benefit: While attention improved BiLSTM baseline (~54% → 63%), it cannot fully compensate for the lack of bidirectional contextual pretraining.
• Efficiency Trade-off: BiLSTM with Attention trains 18-35x faster than transformer models, making it a viable candidate for resource-constrained environments where a 10% accuracy drop is acceptable.

Extension 1

Fine-tuning Strategy Comparison

Three fine-tuning strategies were evaluated on both DistilBERT (67M params) and BERT-base (110M params):

• Freeze backbone: Train only the classification head; backbone weights remain frozen
• Full fine-tune: Train all layers from the start with layerwise differential learning rates
• Hybrid: Train classification head first (5 epochs), then unfreeze backbone with differential LR (15 epochs)

Learning Rate Strategy

A layerwise differential learning rate strategy was applied to both the Hybrid and Full fine-tune approaches to prevent catastrophic forgetting and ensure stable convergence:

• Phase 1 (Warmup): The learning rate is set to 5e-5 for the classifier layer to adapt it to the 20_newsgroups vocabulary.
• Phase 2 (Full Tuning): The classifier remains at 5e-5. The backbone uses a progressive LR: 1e-6 for embedding layers, 2e-6 for initial encoder layers, and 5e-5 for the final layers closest to the output.

Results

Image 8: Training Loss Comparison

Image 9: Validation Accuracy Comparison (Transformers)

Analysis

• The hybrid strategy achieved the best accuracy for both models, outperforming full fine-tuning by ~0.2-0.5% on test accuracy. Training the classification head first provides a stable initialization before updating the pretrained backbone, which leads to slightly better convergence.
• Full fine-tuning performed comparably to hybrid but with slightly lower accuracy, suggesting that updating all layers simultaneously without a warm-up phase can lead to suboptimal adaptation of the classification head.
• Freeze backbone performed significantly worse (~10% lower than hybrid), confirming that the pretrained representations alone are insufficient for this 20-class task and backbone adaptation is necessary.
• Training time scales linearly with model complexity: hybrid takes ~15% longer than full fine-tune due to the additional Phase 1 epochs, but the accuracy gain justifies the cost.

Extension 2

Model Efficiency Comparison

A comparison of model efficiency across DistilBERT and BERT-base, evaluating accuracy vs model size and inference time.

Analysis

• BERT-base provides only a marginal accuracy improvement (+0.89%) over DistilBERT while requiring 1.63x more parameters, 1.82x longer training time, and 2.0x slower inference.
• DistilBERT offers a significantly better accuracy-to-efficiency trade-off: it achieves 98.8% of BERT's accuracy at significantly lower computational cost (1.8x faster training, 2x faster inference).
• This result aligns with the design goal of knowledge distillation — DistilBERT retains ~98% of BERT's performance while being significantly smaller and faster.

Simple Compression: DistilBERT as a Compressed Model

DistilBERT itself serves as a compressed version of BERT-base, produced through knowledge distillation during pretraining:

The experiment demonstrates that model compression via distillation is an effective strategy for reducing model size and inference latency with minimal accuracy loss. For this text classification task, the 39% parameter reduction and 50% inference speedup come at a cost of less than 1% accuracy — a favorable trade-off for most production applications.