ModBERT: a DNA modification and genetic variant-aware DNA large language model

ModBERT, a DNA language model using a five-letter probabilistic genome representation including 5mC, predicts chromatin accessibility in held-out loci and individuals with AUROC 0.88.

• Why it matters: Most DNA LLMs are trained on reference genomes and miss two key regulatory variation sources—genetic variants and DNA methylation—measured together by long-read sequencing.
• What they did: ModBERT was pretrained on soft-target masked language modeling from 242 individuals in the 1000 Genomes Long-Read Sequencing Consortium, then fine-tuned probabilistically to encode one person’s genotype and methylation from a single assay.
• The result: Fine-tuned ModBERT classifies ChIP-seq peaks across 11 histone marks and 69 transcription factors, yields TF sensitivity linked to methylation-aware motif models, and shows variant effect agreement with likely causal caQTLs (Spearman 0.52) rather than linkage-only signals.

AI summary of the abstract · may contain errors · not medical advice · read the paper.