Leakage, Small Data and Strong Baselines: An Open, Reproducible Re-examination of Word2Vec–LSTM ICD Coding

Abstract

Automated ICD coding assigns diagnosis codes to clinical narratives. We re examine an earlier Word2Vec–LSTM ICD-9 coding prototype (NLPK) and replace its unverifiable claims with an open, executable study. First, re-running the pro totype shows that its input vector is computed from the title of the target code itself. A one-timestep LSTM trained on these vectors reaches 0.518 accuracy on the original 80/20 row split, in which 94 patients appear on both sides. A training-free nearest-title lookup reaches 0.770, exactly the fraction of test rows whose code occurs in training. A label-permutation control falls to 0.007. The prototype therefore measures dictionary lookup, not coding. Second, because MIMIC notes require credentialed access, we benchmark on openly downloadable data: the English CodiEsp-D corpus of 1,000 ICD-10-CM-coded clinical case reports, plus Symptom2Disease as a secondary short-text task. Under a single, validation-only protocol for model selection and thresholds, fine-tuning a pre trained clinical encoder in the PLM-ICD style is by far the strongest approach. Over five seeds, Bio_ClinicalBERT reaches micro-F1 0.738±0.012 on CodiEsp 50 and 0.686±0.008 on the top-50 ICD-10 categories, and the 14M-parameter TinyClinicalBERT reaches 0.712±0.012 and 0.627±0.015. The best linear model gets 0.422 on CodiEsp-50 (best macro-F1 0.359), and a wider TF-IDF grid only reaches 0.455. Split, overlap and shuffled-label controls find no leakage behind the gap. Among models trained without a fine-tuned pretrained encoder, TF-IDF linear models remain the best. A CAML-style label-attention CNN comes close (0.377 micro-F1, comparable macro-F1); its paired bootstrap gap to plain TF-IDF+LR has a 95%interval that includes zero. The faithful Word2Vec+BiLSTM reproduction of the original draft (0.308), BiLSTM+LAAT (0.187) and frozen TinyClinical BERT features (0.342) are all significantly worse than TF-IDF+LR. Ablations show that, at this scale, positive-class re-weighting is essential: without it the BiLSTM collapses to the label prior. A bidirectional encoder helps, label attention helps CAML, and Word2Vec initialisation from ICD titles or in-domain text gives no reliable gain over random initialisation. All numbers in this paper come from the released notebook and fine-tuning scripts

Keywords