Vectorized Communication in Distributed LLM Systems: Scenarios, Challenges, and Design Space

Abstract

Distributed LLM systems increasingly exchange intermediate representations across compute, storage, and network boundaries. This survey develops a communication-centered view of these transfers, covering parameters, activations, KV caches, hidden states, and latent messages across distributed training, disaggregated inference, cache pooling, and multi-agent interaction. We organize the literature along four usage perspectives and three fidelity tiers: exact-state reconstruction, lossy compression, and semantic collaboration. The survey relates these settings through task utility, transferred bytes, end-to-end latency, and redundant recomputation. To make this trade-off explicit, we introduce Semantic Density (SD), a task-centered reporting protocol that measures retained utility per normalized transferred byte relative to an uncompressed exact-state reference. Using this framework, we identify recurring challenges in wide-area communication, state reuse, latent-space alignment, and transport--memory coordination, and organize existing approaches around compression, reuse, semantic communication, and storage-aware transfer. We conclude by outlining open problems in alignment, security, fidelity evaluation, standardization, economics, and communication beyond Transformers.

Keywords