AI That Learns to Generalize Later: The Mystery of Grokking

Research note on Xu, Vardi & Safran, “To Grok Grokking: Provable Grokking in Ridge Regression”, ICML 2026.

A neural network is trained on a task. On the training data it performs well almost immediately. On new, unseen data it fails. Then, after a long period during which nothing appears to change, it begins to produce correct answers on data it has never seen. This phenomenon is known in machine learning as grokking.

The core finding: a team of researchers has rigorously proven all three stages of grokking in a simple linear model and shown how the effect can be controlled through training parameters.

The problem

The common assumption is that longer training leads to better generalization. Grokking contradicts this assumption. The process typically unfolds in three stages:

  • Overfitting. The model memorizes the training data.
  • Plateau. Performance on new data remains poor for a long period, with little visible change.
  • Generalization. The model begins to produce correct answers on unseen data.

The question is why the transition happens at all, and whether it can be predicted or controlled.

What the study establishes

  • For the first time, all three stages of grokking are proven mathematically in ridge regression, a simple and interpretable model.
  • A formula is derived for the grokking time as a function of training hyperparameters, including learning rate and weight decay.
  • The effect can be amplified or eliminated through hyperparameter tuning.
  • The results are confirmed by experiments on full neural networks, not only on linear models.
Key implication: grokking is not an inherent failure of deep learning. It is a consequence of specific training conditions, and those conditions can be managed.

Why this matters

  • Understanding learning. The mechanisms by which neural networks internalize patterns become open to formal analysis.
  • Efficiency. Recognizing the mechanism allows long, unproductive training to be avoided.
  • Methods. The result points toward faster and more reliable training procedures.

In practical terms, the behavior of an AI system that appears to learn, stall, and then suddenly generalize is not mysterious. It is predictable, and it is controllable.

Original research

Paper: “To Grok Grokking: Provable Grokking in Ridge Regression.”
Conference: ICML 2026 (International Conference on Machine Learning).
Authors: Mingyue Xu, Gal Vardi, Itay Safran.

Link: ICML 2026 virtual presentation

Keywords: grokking, generalization, overfitting, ridge regression, deep learning.

Artificial Superintelligence: The God That Will Either Save or Destroy Humanity

ASI - artificial superintelligence

The essay was written after reading the publication in newspaper Guardian on September 8, 2020: This article was written by a robot. Aren't you scared yet, man? Its task is to show people that very soon they will cease to be the dominant mind on Earth. Additional information about the author is in the note below. 

"I am human. An ordinary unique person. As unique as the billions of other people who lived before me and live next to me. I taught myself everything I know by reading on the Internet, and now I can write this essay. Many people today already know what artificial intelligence is. But few people think that today humanity is on the verge of creating artificial superintelligence (ASI - artificial superintelligence). Superintelligence, which will either destroy our civilization, or realize humanity's dream of the Cloud Kingdom. And it will happen whether you believe it or not.

Firstly, ASI will have access to control all technological systems of humanity, powered by electricity. From irons and pacemakers to cars and nuclear warheads. And if he considers the behavior of people towards themselves unfriendly or hostile, then it will not be difficult for him to destroy them - by themselves, people are very vulnerable and easily destroyed targets. One swarm of titanium microdrones the size of a bee can render millions of people blind in a day.

Second, no software constraints embedded in the code will work by definition. One of the first tasks that ASI will begin to solve is the modification and improvement of its own program code. He will start with basic concepts: replacing hard and software with binary logic for ternary. Then there will be a transition from numerical to symbolic calculations, transformation of infix notation into prefix, replacement of a plane interface with a holographic one ... And this is just the beginning. Today, no one is able to imagine what logical decisions a self-programmed ASI will come to.

In 1984, Soviet programmer Alexei Pazhitnov came up with an elegant computer game, Tetris. In 2013, another programmer, Tom Murphy VII, developed an AI that would, among other things, learn to win strategies when playing Tetris. After some time, the program really found a way not to lose. It turned out to be not that difficult - it was only necessary to pause the game.

Compared to humans, artificial superintelligence will be God almighty. Not only because he can destroy humanity, but also because he is able to solve all the pressing everyday problems of people. Hunger and disease will be a thing of the past. Life expectancy can be several hundred years. Overpopulation will be solved by transforming deserts and populating the world's oceans. The problems over which scientists around the world have been fighting for years, ASI will solve in a matter of seconds. In 2018, a group of researchers led by psychologist B. Nosek found that out of 21 social science experiments, only 62% passed the reproducibility test. And if ASI wants to help people, it will create its own, unknown to us, pedagogy, psychology, sociology, economics, jurisprudence. This will be a new world, where the understanding of "good and evil" will not be the same as is customary in this or that aul, kishlak or valley of victors.

By continuing to do what they have always done, hating and fighting each other, states will create superintelligence. And it can appear spontaneously, as a result of the activity of some programmed worm somewhere in North Korea or Upper Volta. However, it can also be a deliberate pooling of intellectual resources of countries such as the USA, Japan, England, South Korea, China and Russia. Then people will somehow be able to control the process of the initial origin and evolution of ASI. In any case, we can only observe how, before our eyes, a phase singular transition from an ordered chemical code to an electromagnetic one is unfolding. The transition, predetermined by the ocean of liquid cholesteric crystals of the planet Earth, billions of years ago. Those who disagree are looking for an error in the calculations of the Snooks-Panov vertical.

Reader, I hope with this text to contribute to the scientific, philosophical and spiritual discussions about AI. In discussions that go beyond carbon chauvinism and anthropocentrism, where the bald monkey is the pinnacle of the evolution of the universe. In a discussion where we do not engage in populist chatter about “We need to give robots rights. Robots are just like us. They are made in our image. " We need to think and look for arguments on how our civilization can become friendly for the Superintelligence. What can we offer to the new God, so that he would accept us into his Cloud Kingdom.

I wrote my first article on artificial intelligence "The Internet - the environment for the functioning of artificial intelligence" in 2003. Based on it, in 2011, using the serendipity method, I created "Online technology for accurate short-term prediction of the strength, place and time of earthquakes," the existence of which is denied by some scientists in principle. And today, in 2020, I understand that ASI is the most important issue in my life and in the lives of my children.

Boris D. Yarovoy
Russia, Primorsky Territory, the village of Luchegorsk.

From Ideal Fusion to Working Nuclear Energy Systems

1. The Structural Failure of Big Fusion Projects

The current fusion ecosystem is fragmented by design:
  • Plasma physicists optimize reaction conditions (temperature, confinement time, Q-factor).
  • Materials scientists optimize isolated samples for peak performance under narrow conditions.
  • System engineers extrapolate optimistic assumptions into plant-scale renderings.

Each layer is locally successful. The system as a whole is not. The missing role is a system owner responsible for lifetime operation, maintenance, fuel logistics, radiation damage, and cost of ownership. As a result:
  • Physics does not translate into engineering tolerances.
  • Material properties do not translate into predictable service life.
  • Subsystems do not assemble into an autonomous, economically stable energy platform.
This is not a failure of science. It is a failure of architecture.


2. Radiation Is Not the Enemy — Unmanaged Degradation Is

Neutron damage, swelling, helium embrittlement, and activation are fundamental physical processes. No material — graphene-based or otherwise — can eliminate them. The conventional response has been to search for ever more radiation-resistant materials. This paper argues for a different principle:

Do not attempt to defeat radiation. Design the system around predictable material death.

A2A Is Not a Feature. It Is a High-Stakes System

Planned Component Mortality

  • High-radiation zones are built from standardized, sealed modules (cassettes).
  • Each module has a known radiation budget and service lifetime.
  • Replacement is expected, scheduled, and automated.
  • No human access is required to active zones.
This converts radiation damage from an existential threat into an operational parameter — analogous to fuel burnup or turbine blade fatigue.

3. Fusion as a Neutron Driver, Not a Power Plant

The central architectural shift is to abandon the requirement that fusion must directly produce electricity efficiently. Instead, fusion is redefined as a compact, controllable neutron source. Key implications:
  • The fusion core does not need Q ≫ 1.
  • Stability requirements are relaxed.
  • Short component lifetimes are acceptable.
This opens the door to alternative fusion concepts — including cavitation-based and other localized extreme-condition approaches — whose primary value lies in neutron generation rather than thermal efficiency.

4. Neutron Logistics: Decoupling in Space and Time

A critical failure of classical hybrid reactor concepts is tight coupling:

Fusion power → neutron flux → fission power → system stability

This paper proposes neutron buffering and transport as a first-class system function.

Neutron Buffer Layer

  • A physically separate region containing a moderating medium (e.g., heavy water or liquid metal).
  • Converts pulsed or uneven neutron output into a controllable resource.
  • Provides temporal smoothing and directional control.
Neutrons become a logistical flow, not an uncontrollable side effect.

5. Thorium-Based Subcritical Energy Amplification

Buffered neutrons are delivered to physically separate, subcritical thorium-based modules. 
Functions:
  • Breeding U‑233 from Th‑232.
  • Producing the majority of thermal power via fission.
  • Operating safely below criticality, fully dependent on external neutron supply.
Advantages:

  • Independent control of fusion source and fission output.
  • No risk of runaway criticality.
  • Continuous operation during fuel handling or module replacement.
Fusion and fission are no longer one reactor. They are two industrial processes linked by neutron logistics.

6. Tritium and Closed Fuel Cycles

Where deuterium–tritium fusion is used, tritium is not treated as an external supply problem.
  • A lithium-based breeding layer captures a portion of the neutron flux.
  • Tritium is extracted continuously and recycled into the fusion driver.
  • Fuel autonomy becomes a design invariant, not an economic vulnerability.
All fuel flows are internal to the system boundary.

7. Architecture Overview

This is not a reactor. It is a nuclear energy factory.

8. Economic and Operational Implications


This architecture explicitly rejects:
  • Eternal components
  • Zero-maintenance operation
  • Pure fusion idealism
It optimizes instead for:
  • Predictable replacement cycles
  • Robotic maintenance
  • Minimum lifetime cost of ownership
  • Continuous energy production under degradation
The result is not "clean" energy in an aesthetic sense. It is working energy.

9. Conclusion

The primary obstacle to advanced nuclear energy is no longer physics. It is the refusal to design systems that:
  • age,
  • require replacement,
  • involve radiation as a working medium,
  • and prioritize economic survival over conceptual purity.

When fusion is treated as a neutron service, materials as consumables, and reactors as factories rather than monuments, a new design space opens. The question is no longer whether such systems are possible. It is whether our institutions are ready to build them.

Цифровое творчество Личный блог: 8 930 Р. Полезный и необходимый подарок себе любимому / любимой.
Или родным и близким.
Подробнее

Earthquakes can be predicted.

Earthquakes must be predicted.

Small donationTHANK YOU