IB Computer Science EE

February 2025

Evaluated the efficacy (compression size and accuracy retention) of magnitude-based weight pruning for image classification models.

Summary of findings

• Magnitude-based weight pruning can reduce model size in dense neural networks to a notable extent, with limited reduction in performance accuracy. Experiment results indicate that model parameter count can be compressed between 33-100 times on MNIST and 20-50 times on CIFAR-10 and CIFAR-100, and model size can be compressed by approximately 6-10 times without more than 10% reduction in relative performance accuracy.
• There are underlying complexities to compression effectiveness. For example, additional fine-tuning must occur after pruning to achieve the best compression performance, implying the necessity for additional computing resources and time to support the compression process.
• Compression effectiveness appears to decrease as dataset complexity increases. As such, weight pruning may be insufficient for the deployment of extremely large models for complex tasks.
• On complex datasets, compressed models that retain performance remain overparameterized. As such, model compression through weight-pruning likely cannot transcend fundamental constraints of minimum model size arising from its training dataset’s complexity level. Nonetheless, model compression can help reduce redundant training by allowing vendors to train a single large model and compress it to various sizes, instead of training several models of different sizes. It also enables the repurposing of larger general-purpose models for more specialized tasks and limited environments. As such, model compression still has practical applications and significant implications for real-world model deployment.

Key graphs