Hiring in life sciences? Share your open positions with our professional community. Read more Close

Advertisement

Revisiting dynamic batch scheduling for grokking: the role of batch size, sample budget, and regularization.

Created on 17 Nov 2026

Authors

Roland Fang

Published in

Frontiers in artificial intelligence. Volume 9. Pages 1860893. Epub Nov 02, 2026.

Abstract

Grokking-the sudden generalization of neural networks long after overfitting-has inspired training strategies aimed at accelerating its onset. This paper revisits dynamic batch size scheduling and asks whether larger batches provide intrinsically better optimization dynamics, or whether they appear better because they process more examples per update. We evaluate three dynamic schedules (linear, exponential, and step-wise increases from 32 to 512) against fixed batch sizes of 32 and 512 across four algorithmic tasks, each with 10 random seeds. When measured by update steps, the fixed-512 baseline is highly competitive: on modular division it reaches 99.6% final accuracy with 10/10 runs grokking at a mean of 3,245 steps. However, this advantage changes substantially after normalizing by total examples processed. Under a matched budget of 240,000 training examples (equivalently 960,000 input tokens), fixed-512 performs poorly and never groks on any task, while smaller-batch and dynamic strategies remain competitive on the modular tasks. A focused hyperparameter sensitivity study additionally shows that grokking depends strongly on weight decay, model size, and maximum batch size: without weight decay, none of the tested strategies grok, while intermediate or strong weight decay can make both fixed-large and dynamic strategies reliable. These results show that larger batches improve update-step efficiency but are not necessarily more sample-efficient, and that their effect is conditional on regularization and capacity. We therefore revise the original interpretation: batch size magnitude is a major confound in grokking experiments, and claims about acceleration should report both update-step and sample-normalized metrics.

PMID:
43041375
Bibliographic data and abstract were imported from PubMed on 17 Nov 2026.

Read full publication at:
Please sign in to see all details.

Advertisement

Stats

  • Community rating n/a 0 votes
  • Reviewers' rating n/a 0 votes
  • Your rating

1-terrible, 9-excellent. How would you rate this publication? Sign in in to submit your rating.

  • Recommendations n/a n/a positive of 0 vote(s)
  • Views 8
  • Comments 0

Recommended by

  • No recommendations yet.

Post a comment

You need to be signed in to post comments. You can sign in here.

Comments

There are no comments yet.

Advertisement