Post
mr. TIM
timkellogg.me
did:plc:ckaz32jwl6t2cno6fmuw2nhn
AttentionInfluence: for pretraining data selection
Good data matters, but how do you find it?
This paper uses the attention heads from existing models to calculate & rank how valuable the data will be during training
Mask out critical heads and calculate the loss
arxiv.org/abs/2505.07293
2025-05-14T03:02:26.847Z