Teresa Delgado, Hugo Hazard, Martin Benfeghoul, Jun Wang, Zafeirios Fountas · KV Cache Compression

LowRAM: Low-Rank Keys and Amortised Values for Composable KV Cache Compression at Test-Time

LowRAM is a KV cache compression method for autoregressive decoding that pairs test-time value cache amortisation with cross-layer low-rank key compression. The authors report 4x near-lossless value compression, up to 5x higher peak decode throughput at 4x compression, and 26.9x compression at 94.5% of baseline performance on long contexts when combined with quantisation and eviction. Results are reported across three model families.