LLCBench: Benchmarking Lossless Compression of Large Language Models

Abstract

The rapid proliferation of Large Language Models (LLMs) has posed significant challenges to model storage, distribution, and inference. While various reviews and benchmarks for lossy compression like distillation, pruning, and quantization have emerged to address these issues, their inherent lack of data integrity restricts their utility in scenarios requiring high-fidelity persistent backup and the reliable sharing of LLM repositories. To this end, we present the first review and benchmark of lossless compression for LLMs, serving as a further complement to existing lossy work. Specifically, we first propose a classification methodology for LLM lossless compression, including Floating-Point-Driven (FPMD), Large-Language-Model-Driven (LLMD), Learning-Based Lossless Compression (LLC), and Traditional Lossless Compression (TLC). Following this, we conduct a detailed review of these four types of methods and discuss their advantages and shortcomings. Subsequently, we evaluate the lossless compression performance, throughput, and resource consumption of 24 solutions across 10 real-world LLMs using our benchmark. This comprehensive evaluation covers 10 critical metrics and spans over 5,700 hours of CPU/GPU computation time. Furthermore, drawing upon our systematic review and benchmark analysis, we identify several promising future research directions, including automated parameter optimization, intelligent storage and distribution frameworks, parallel acceleration for compression algorithms, and the development of robust algorithmic designs. We organize all results and scripts as the LLMs Lossless Compression Benchmark (LLCBench), which can be found at [https://github.com/fahaihi/LLCBench](https://github.com/fahaihi/LLCBench).

Keywords