The abstract is read at the publisher; the summary is JClub's.
Using an LLM as the primary coder for 6,982 Instagram and TikTok posts, per-variable reliability was verified empirically, with human–LLM agreement reaching 0.847 (N=150) and 0.868 (N=50).
- Why it matters: Health content analysis lacks reporting standards for LLM-assisted coding, creating uncertainty about reliability, error direction, and how to handle failures across variables.
- What they did: They built a six-part verification framework: human-authored and calibrated instruments, quote-anchored coding, and blinded verification in both representative (N=150) and outcome-enriched (N=50) samples, then reported per-variable agreement using Gwets AC1.
- The result: After revising two ambiguous rules, human–human agreement on the primary outcome rose from AC1 0.456 to 0.977; five of eight conditional variables fell below 0.80, and misinformation prevalence was likely underestimated because humans flagged it more often than the LLM.
The findingWhy it mattersWhat they didThe result
AI summary of the abstract · may contain errors · not medical advice · read the paper.