A neural network is trained on a task. On the training data it performs well almost immediately. On new, unseen data it fails. Then, after a long period during which nothing appears to change, it begins to produce correct answers on data it has never seen. This phenomenon is known in machine learning as grokking.
The problem
The common assumption is that longer training leads to better generalization. Grokking contradicts this assumption. The process typically unfolds in three stages:
- Overfitting. The model memorizes the training data.
- Plateau. Performance on new data remains poor for a long period, with little visible change.
- Generalization. The model begins to produce correct answers on unseen data.
The question is why the transition happens at all, and whether it can be predicted or controlled.
What the study establishes
- For the first time, all three stages of grokking are proven mathematically in ridge regression, a simple and interpretable model.
- A formula is derived for the grokking time as a function of training hyperparameters, including learning rate and weight decay.
- The effect can be amplified or eliminated through hyperparameter tuning.
- The results are confirmed by experiments on full neural networks, not only on linear models.
Why this matters
- Understanding learning. The mechanisms by which neural networks internalize patterns become open to formal analysis.
- Efficiency. Recognizing the mechanism allows long, unproductive training to be avoided.
- Methods. The result points toward faster and more reliable training procedures.
In practical terms, the behavior of an AI system that appears to learn, stall, and then suddenly generalize is not mysterious. It is predictable, and it is controllable.
Original research
Paper: “To Grok Grokking: Provable Grokking in Ridge Regression.”
Conference: ICML 2026 (International Conference on Machine Learning).
Authors: Mingyue Xu, Gal Vardi, Itay Safran.
Link: ICML 2026 virtual presentation
Keywords: grokking, generalization, overfitting, ridge regression, deep learning.
Комментариев нет:
Отправить комментарий
Спасибо за комментарий. Thank you for comment. 谢谢你的评论