KeyMem: Graph-Augmented Vector Retrieval for Persistent Conversational Memory in LLM Agents

Abstract

Long-term memory is a critical capability gap in LLM-based agents. Existing approaches either compress conversation history into structured facts(sacrificing information fidelity) or rely on pure vector retrieval that fails on multi-hop and temporal reasoning questions. This paper presents KeyMem, a persistent memory system that stores raw conversational turns in a knowledge graph and retrieves them through a dual-path architecture combining keyword vector search with graph traversal. The system introduces three key contributions: (1) a Fragment-based multi-hop expansion mechanism that aggregates topically related memories for complex queries; (2) a source-aware scoring formula with query-local normalization that requires no preset hyperparameters; and (3) a lightweight reference resolution state machine for anaphora disambiguation during storage. On the LoCoMo-10 benchmark, KeyMem achieves an overall token F1 of 0.609 and Recall of 0.847, outperforming mem0 (F1=0.372) by 63.7% while preserving full information fidelity through raw turn storage. The code is available at https://github.com/g0dA/KeyMem.

Keywords