Efficient and accurate steering of large language models through attention-guided feature learning.

Attention-guided activation steering nearly doubles the number of successfully steerable concepts across models up to 70 billion parameters.

• Why it matters: Understanding and controlling how semantic concepts are stored in large language models is crucial for improving their interpretability and capabilities, but existing methods are fragile and inconsistent.
• What they did: The authors developed an attention-guided framework that automatically selects relevant token embeddings, accounts for feature heterogeneity, and identifies key layers, tested on 512 semantic concepts across various model sizes.
• The result: The framework significantly outperforms previous methods, enabling more reliable concept steering and revealing the distribution of concept features across layers, thus supporting scalable fine-tuning of industry-scale LLMs.