.

AI That Learns to Generalize Later: The Mystery of Grokking

Research note on Xu, Vardi & Safran, “To Grok Grokking: Provable Grokking in Ridge Regression”, ICML 2026.

A neural network is trained on a task. On the training data it performs well almost immediately. On new, unseen data it fails. Then, after a long period during which nothing appears to change, it begins to produce correct answers on data it has never seen. This phenomenon is known in machine learning as grokking.

The core finding: a team of researchers has rigorously proven all three stages of grokking in a simple linear model and shown how the effect can be controlled through training parameters.

The problem

The common assumption is that longer training leads to better generalization. Grokking contradicts this assumption. The process typically unfolds in three stages:

  • Overfitting. The model memorizes the training data.
  • Plateau. Performance on new data remains poor for a long period, with little visible change.
  • Generalization. The model begins to produce correct answers on unseen data.

The question is why the transition happens at all, and whether it can be predicted or controlled.

What the study establishes

  • For the first time, all three stages of grokking are proven mathematically in ridge regression, a simple and interpretable model.
  • A formula is derived for the grokking time as a function of training hyperparameters, including learning rate and weight decay.
  • The effect can be amplified or eliminated through hyperparameter tuning.
  • The results are confirmed by experiments on full neural networks, not only on linear models.
Key implication: grokking is not an inherent failure of deep learning. It is a consequence of specific training conditions, and those conditions can be managed.

Why this matters

  • Understanding learning. The mechanisms by which neural networks internalize patterns become open to formal analysis.
  • Efficiency. Recognizing the mechanism allows long, unproductive training to be avoided.
  • Methods. The result points toward faster and more reliable training procedures.

In practical terms, the behavior of an AI system that appears to learn, stall, and then suddenly generalize is not mysterious. It is predictable, and it is controllable.

Original research

Paper: “To Grok Grokking: Provable Grokking in Ridge Regression.”
Conference: ICML 2026 (International Conference on Machine Learning).
Authors: Mingyue Xu, Gal Vardi, Itay Safran.

Link: ICML 2026 virtual presentation

Keywords: grokking, generalization, overfitting, ridge regression, deep learning.

Комментариев нет:

Отправить комментарий

Спасибо за комментарий. Thank you for comment. 谢谢你的评论