Training and Deploying Machine Learning Services under the Neural Scaling Law

A machine-learning service, from a fraud model to a frontier language-model product, gets better as its provider invests more in data, compute, and retraining, but each round of investment helps less than the last. The improvement follows a neural scaling law: after $n$ periods of investment, the model's prediction error is $v(n) = a\,n^{-b}$, where $a > 0$ is the error scale (how poor the model is at the start) and $b > 0$ is the learning rate (how quickly the error falls). Quality keeps rising while its marginal value shrinks. The service runs for a finite horizon of $T$ periods; the provider pays a cost $c$ for every period it keeps investing and must decide when to stop and run the service with the model it has. This is an optimal stopping problem.

When $a$ and $b$ are known, a continuous relaxation of the problem is strictly concave and stays within a constant of the discrete optimum, independent of the horizon $T$. The optimal investment period then grows only as $\Theta\bigl(T^{1/(b+1)}\bigr)$, that is, in proportion to $T^{1/(b+1)}$, a vanishing share of the horizon: even when long-run value scales with the horizon, diminishing returns eventually stop justifying the cost. Slower learning (smaller $b$) or a worse starting model (larger $a$) shift the stopping time in predictable ways.

In practice $a$ and $b$ are unknown and must be learned from noisy performance measurements while deciding. The proposed rule refits the scaling curve after every period, builds a confidence region for the pair $(a, b)$, and keeps investing only while the most optimistic parameters still consistent with the data say more learning is worth it. Its expected regret grows at most in proportion to $\log^3 T$, written $O(\log^3 T)$.

• The continuous relaxation is faithful: its optimal value differs from the discrete problem's by a constant that does not grow with the horizon $T$.
• The optimal investment period scales as $\Theta\bigl(T^{1/(b+1)}\bigr)$, with comparative statics in the learning rate $b$ and the error scale $a$.
• With $a$ and $b$ unknown, the optimistic stopping rule has expected regret $O(\log^3 T)$ relative to the full-information optimum.