When stiffness and entropy constraints backfire in deep learning

Many problems in deep learning can be described as propagation problems. As a network grows deeper, hidden states and gradients pass through more layers, giving local errors more chances to grow through repeated composition. Exploding gradients, vanishing gradients, oscillation between layers, feature concentration and representation collapse look like different phenomena. They share one question underneath: can information pass through a deep network in a stable form?

That question led to FE-E, short for Finite Element-Entropy. The method treats network depth as a one-dimensional mesh and the gradient at each layer as a vector field at a mesh node. It then borrows stiffness, mass and information entropy from finite-element analysis to constrain propagation. The idea is formally natural. The stiffness term suppresses abrupt changes between neighboring layers. The mass term constrains total gradient energy. Entropy prevents that energy from concentrating in a small number of layers or feature directions.

From the equations alone, FE-E appears likely to make training more stable and efficient. The experiment produced a more interesting result. A constraint can improve the local shape of propagation without improving learning. Too much stability can even delay the phase transition in which learning actually occurs.

What the constraints were meant to do

Let g_l be the backward gradient at layer l. The finite-element stiffness energy is:

E_stiff = sum_l ||g_(l+1) - g_l||^2 / Delta z

It penalizes sharp changes in the gradient across layers. The energy rises when a gradient suddenly becomes larger at one layer, rotates rapidly or oscillates at high frequency along network depth.

The mass energy is:

E_g = G^T (M tensor I) G

It measures total gradient energy along depth. Keeping it near a reference value can suppress overall explosion or decay.

The depth-wise entropy begins with the gradient-energy share:

p_l = w_l ||g_l||^2 / [sum_j w_j ||g_j||^2 + epsilon]

and is defined as:

H_g = -sum_l p_l log(p_l + epsilon)

Low entropy means that the gradient is concentrated in a small number of layers. Very high entropy may mean that every layer is being forced into nearly the same role. The original FE-E proposal was to keep entropy inside a suitable interval while stabilizing gradient magnitude and continuity across layers.

That proposal hides an assumption that is easy to miss: smoother propagation, steadier energy and a more even distribution must produce better learning. The experiments show that this assumption does not always hold.

The experiment ran in the other direction

I ran an MLX experiment with 128 layers, width 32 and four attention heads. The baseline used AdamW with Gradient Smoothing. Each FE-E intervention was limited to 5 percent of the task-gradient norm and applied at frozen frequencies of 1, 3 and 5 percent. The experiment covered four environments: no propagation noise, high-frequency perturbations between layers, globally amplified residual energy and energy concentrated in the middle layers. The endpoint was token accuracy of at least 99 percent on three consecutive checks.

Across 12 paired comparisons between an FE-E dose and the SHAM baseline in the same environment, only three FE-E runs reached the endpoint earlier. The other nine were slower. In the normal environment, plain GS was confirmed at step 1,536. FE-E at 1, 3 and 5 percent needed 1,600, 1,632 and 1,888 steps. With no clear propagation abnormality, every tested dose made the optimization worse.

The middle-layer concentration environment produced one local positive result. GS needed 1,952 steps, while 3 percent FE-E needed 1,824, an improvement of 128 steps. It was still slower than GS in the normal environment at 1,536 steps. It was also a single-seed result selected from 12 paired comparisons, so it is not evidence of an "optimal intervention rate."

The globally amplified energy environment was more revealing. Before the phase transition, the training-loss slope for plain GS was about -0.00062 per 100 steps. At 5 percent FE-E it was about -0.01386, which looked much faster. Yet GS confirmed 99 percent at step 1,440, while 5 percent FE-E did not get there until step 1,600. The local loss slope became much steeper, and final convergence arrived 160 steps later.

The computational cost also matters. In this small model, the median FE-E intervention step took about 2.24 times as long as a normal step. Across experiments, the average cosine between the FE-E gradient and the task gradient was negative. FE-E was usually not moving farther in the task direction. It was spending some of the optimizer's capacity resisting the task gradient in exchange for a more orderly propagation pattern.

Why a steeper slope did not mean earlier convergence

The time needed to reach 99 percent accuracy can be roughly decomposed as:

T_99 = T_plateau + T_transition + T_confirm

T_plateau is the time the model spends at low accuracy. T_transition is the jump from low to high accuracy once the transition begins. T_confirm comes from consecutive validation checks. A short-term loss slope mainly describes local descent around the transition. It does not say when the model entered the transition.

In the globally amplified energy experiment, 5 percent FE-E did narrow the 10-to-90-percent transition from 96 steps to 64. But it pushed the first sustained crossing of 10 percent from step 1,280 to step 1,440. The model waited 160 extra steps before it began to learn in earnest. Saving 32 steps inside the transition could not make up for that delay.

This exposes the ambiguity in "falling faster." A model may move faster after it has found a useful direction while taking longer to find that direction. It may also lower cross-entropy by raising confidence on easy tokens or tokens it already predicts correctly, without pushing the remaining difficult tokens across a decision boundary. Continuous loss and a discrete phase change in accuracy are not the same measurement.

The short-term slope in this experiment has another methodological limit. It was calculated retrospectively inside each trajectory's own pre-transition window, not as a causal difference immediately before and after an FE-E intervention at the same model state. Different methods therefore refer to different absolute time ranges. The slope describes a trajectory. It should not be used directly as a gating threshold.

Why stiffness and entropy can work against learning

The stiffness term suppresses high-frequency changes across layers, but high frequency is not automatically noise. Different layers perform different functions. A large update in a small set of layers may be necessary to form a new representation, redistribute functions or cross an optimization barrier. A strong stiffness constraint can misclassify necessary structural reorganization as propagation instability.

Entropy has the same double edge. A concentrated gradient may mean that a few layers are carrying all the learning pressure. It may also mean that the model is developing useful specialization. Entropy can detect concentration. Concentration alone does not tell us whether we are seeing collapse or productive focus.

Mass energy stabilizes the overall magnitude, but learning may not require constant energy. Some phase changes may depend on a brief gradient pulse. If the reference energy cannot distinguish a harmful explosion from a useful jump, the mass term suppresses both.

FE-E is therefore not a neutral mathematical repair. It has structural preferences: continuity, stability and limited concentration. A constraint can produce a net gain only when the task has an abnormality that violates one of those preferences and that violation is actually harmful. Otherwise, the method pushes the model toward a tidier state that may not be better at learning.

From controller to observer

None of this makes stiffness, mass and entropy useless. It suggests a different role. Instead of adding FE-E to the gradient by default, we can first use these quantities to observe propagation:

R_stiff = sum_l ||g_(l+1) - g_l||^2 / [sum_l ||g_l||^2 + epsilon]

E_g = sum_l w_l ||g_l||^2

H_g = -sum_l p_l log(p_l + epsilon)

Normalized stiffness describes abrupt changes between layers. Mass energy detects overall explosion or decay. Depth-wise entropy detects persistent concentration. These measurements can be made with stopped gradients, without computing higher-order derivatives of the FE-E regularizer with respect to the parameters. That reduces both cost and interference.

An observer has to answer something stricter than whether a number looks unusual. Does the state persist? Can it predict worsening loss, uncontrolled gradients or a delayed transition tens of steps ahead? What is its false-positive rate? Intervention is justified only when the indicator has forward predictive value.

Different abnormalities should not trigger the same response. An energy explosion can first be met with gradient clipping or a lower learning rate. High-frequency jumps may call for temporary smoothing. Only persistent concentration of energy across layers might enter a trial window for a small FE-E dose. If validation performance, the margin on difficult tokens or signs of an approaching transition do not improve, the intervention should stop or roll back. FE-E then changes from a controller that continually corrects the model into a sensor that asks whether the model needs help.

From deep networks to cognition and education

If a Transformer is compared with a multistage cognitive process, the FE-E experiment also offers an educational analogy. This does not mean that network layers correspond to brain regions or that an artificial gradient is neural activity. It treats both as processes in which information passes through several stages and gradually forms structure under limited resources.

In this analogy, training loss resembles the current error rate in practice. The short-term loss slope resembles the speed at which performance becomes more fluent. The phase change in accuracy is closer to the moment when a rule is understood and a transferable concept forms. A stiffness constraint resembles a demand that cognitive steps stay continuous and orderly. Entropy control prevents attention and strategy from becoming too concentrated. An FE-E intervention resembles a teacher's hint, scaffolding or metacognitive control.

The experiment's backfire then has a clear educational meaning. A smoother learning process does not prove that understanding arrived earlier. Frequent hints, fixed procedures and repeated exercises may reduce immediate errors while shrinking the space for trial and error, strategy changes and conceptual reorganization. A student can look more fluent without transferring the idea once the hints disappear or the problem changes.

High-frequency variation is not always harmful either. Hesitation, repeated attempts and jumps between strategies can be part of exploring a new structure rather than signs of cognitive instability. The hard part of educational intervention is not eliminating variation. It is distinguishing useful exploration from being persistently stuck.

The educational extension of FE-E should therefore be a metacognitive observation framework, not continuous discipline. It would watch whether errors cluster around one concept, cognitive load stays high for too long, strategies collapse into a single route, or difficulty persists long enough to show that help is needed. An intervention should not be judged only by whether the current problem is answered more quickly. It should be judged by unaided mastery, transfer, delayed retention and dependence on hints.

A testable experiment could compare fixed-frequency hints, no hints and adaptive hints triggered by an observer. If adaptive hints improve only the practice-period slope without improving transfer or delayed retention, they reproduce the optimizer's mistake. The framework has educational value only if it shortens the time to real mastery while reducing dependence on help.

What the failed optimizer left behind

FE-E began as an attempt to stabilize gradient propagation with finite elements and prevent information concentration with entropy. The experiment did not support the strong claim that it works as a general optimizer. It exposed a broader problem instead: any stability constraint can mistake a useful nonstationary process for a fault.

Stability is not another word for learning. Smoothness is not sufficient evidence of understanding. That may be true for deep networks and educational intervention alike. The question is not how to keep every local trajectory orderly. It is which variation is damaging learning and which variation is producing a new structure.

The value of stiffness, energy and entropy may lie less in deciding how a system should change than in helping us see how it is changing. Once a constraint becomes an observation rather than an order, the question changes from "How do we remove instability?" to "When should we intervene, and when should we allow exploration?"