I wrote this post because I was frustrated to find no convincing theoretical explanation of the success of the Adam optimizer (Kingma, 2014). More precisely, Adam is the RMSprop optimizer (Hinton, 2012) + momentum. While understanding momentum is quite simple as a way to deal with a badly conditioned loss landscape, the RMSProp update rule is often unintuitive when coming from the world of quadratic optimization; in particular, the square root in the denominator is quite intriguing. Fortunately, I stumbled on an article that gave a somewhat satisfying answer (Aitchison, 2020). However, I found the derivation somewhat convoluted, particularly the introduction of Ornstein–Uhlenbeck dynamics. In this article, I describe a Bayesian derivation of the RMSProp update rule, close to the one proposed in the mentioned article, but conceptually simpler.

The derivation uses the framework of Bayesian estimation/filtering, unlike other more common geometric approaches.

Let's begin! We will first demonstrate how SGD (Stochastic Gradient Descent) can be derived from Bayesian principles and then show how Adam refines it.

Bayesian derivation

Maximum likelihood principle

Usually, we train models using the maximum likelihood principle: we want to find the most likely parameters $\v{\theta}^*$ of our model given the observation of our dataset $\mathcal{D}$. More formally, we are searching:

Bayes theorem is usually used here to get:

And using the log likelihood instead of the likelihood, we get:

This motivates the use of the loss function as an optimization objective with regularization of the weights (e.g., weight decay) to model the evidence and prior terms, respectively. The analysis generally proceeds by introducing an optimizer to find the best parameters using geometric arguments. It then explains the difficulty of obtaining the true gradient and introduces Stochastic Gradient Descent (SGD) as a solution to this issue.

Going further to discover SGD

In this section, we further apply Bayesian methods to gain deeper insight into the problem.

In practice, the model sees dataset samples sequentially. At each step, we present the model only a subset $\mathcal{B}$ of the dataset (a "batch"). Suppose we are already at some step $t$ of our training and we have an estimate $\v{\theta}_t$ of $\v{\theta}^*$. We have a new batch $\mathcal{B}$, and we want to update our estimate using the information it contains. As before, we can express this problem as a maximum likelihood estimation problem. However, this time we will not assume we know the whole dataset, only the previous parameter estimate and the new batch:

We can use the Bayes theorem here too:

Taking the log likelihood and getting rid of constant (wrt $\v{\theta}$) terms, we get:

Now, if we decide to model the term $p(\v{\theta}|\v{\theta}_t)$ by a normal distribution $\mathcal{N}(\v{\theta}_t, \sigma^2 \v{I})$ with a variance $\sigma^2$ chosen to represent our uncertainty about the value of $\v{\theta}^*$ prior to observing $\mathcal{B}$, we get:

The second term forces the optimal value to be relatively close to $\v{\theta}_t$. We can thus use a linear approximation for the loss function $\mathcal{L}(\theta) = -\log{p(\mathcal{B} | \v{\theta})}$ near $\v{\theta}_t$, the problem becomes the following minimization problem:

This is the stochastic gradient descent algorithm. This derivation shows that the SGD algorithm can be understood and derived using Bayesian principles. The learning rate represents the variance in our belief of the "optimal parameters". As a bonus, within this framework, decaying learning rate scheduling has a clear interpretation as an increase in confidence in the estimate of the best parameters as training progresses. A learning rate decreasing like $1/t$ would match the Cramer-Rao bound convergence rate.

Going even further to discover RMSProp

The question is, can we derive other popular optimization algorithms from this framework? In particular, can we estimate $\sigma^2$ instead of guessing its value?

We will start from here:

In our derivation of SGD, we used a linear approximation for $\mathcal{L}(\v{\theta}) = -\log p(\mathcal{B}|\v{\theta})$. We will see that we can actually do better using a quadratic approximation. Indeed, the Taylor expansion theorem states that we can do the following approximation:

with:

• $\v{g} \triangleq \nabla \log p(\mathcal{B}|\v{\theta})$
• $\v{H} \triangleq \frac{\partial^2}{\partial \v{\theta}^2} \log p(\mathcal{B}|\v{\theta})$

Unfortunately, computing $\H$ is intractable when there are many parameters. Fortunately, we will see that we can get an estimate of $\mathbb{E}[\v{H}]$. indeed:

Defining $\v{G}\triangleq \mathbb{E}\left[\v{g}\v{g}^\top\right]$, our approximation becomes:

and

With $\v{\theta^*} = \v{\theta}_t - \v{G}^{-1} \v{g}$ (the proof is simple and left to the reader or to ChatGPT). It means that $p(\mathcal{B}|\v{\theta})$ is approximately Gaussian.

It follows that $p(\v{\theta}|\mathcal{B}, \v{\theta}_t) = \frac{p(\mathcal{B}|\v{\theta}) p(\v{\theta}|\v{\theta}_t)} {p(\mathcal{B}|\v{\theta}_t)}$ is also close to a Gaussian (as a product of Gaussian) with inverse covariance:

with $\v{\Sigma}_t$ the covariance matrix of $p(\v{\theta}|\v{\theta}_t)$. (To see it, multiply the two terms in the numerator; the inverse covariances will sum in the exponential).

And mean:

To account for our approximation errors, we can add a small term $\eta^2 \v{I}$ to the covariance:

This modelization (quadratic approximation + added variance) defines a new Bayesian optimizer:

However, even in this form, the approximation is not usable because the matrix is generally too big. But if we suppose that the gradients are approximately centered and uncorrelated, off-diagonal terms vanish, and we can estimate $\v{G}$ as a diagonal matrix:

the algorithm can now be decomposed component wise by denoting $\v{s}$ the estimated vector containing the diagonal element of $\v{G}$ obtained by an exponential moving average and $\v{\sigma}$ the diagonal of $\v{\Sigma}$:

Linking the new optimizer to Adam

You may not recognise its proximity to the RMSProp update rule. Let's push the analysis a bit further to see how it is connected.

To see to what value the covariance matrix converges, we can search for a fixed point verifying:

In our simplified optimizer, we replaced $\v{G}$ by $\v{s}$ and $\v{\Sigma}$ by $\v{\sigma}$, and all the operations are element-wise. We can solve this equation easily using scalar arithmetics:

This a second order polynomial in $\v{\sigma}^2$ let's compute the determinant:

And the unique positive solution:

When $\eta$ is small enough (our quadratic approximation is good), we can neglect the term $\eta^2\v{s} / 4$ before $1$ leading to the following approximation:

Again, when $\eta$ is small, we can neglect the first term. Finally, we get:

Plugging this into our parameter update rule, we get the following approximation of the Bayesian optimizer described before:

This is the RMSProp optimizer update. It has a benefit over the full Bayesian optimizer: it needs one less statistic to be stored, reducing memory usage.

Choosing $\eta$

The parameter $\eta$ is introduced to account for the fact that we are using approximations which reduce our confidence in the estimates. It's interesting to study the optimizer in different regimes.

Large $\eta$

When $\eta$ is large (i.e., we have little confidence in our approximation) and $\eta^2 \v{s} / 4 \gg 1$, we can make the following approximation:

and the update rule is:

This is the SGD algorithm. Thus, both RMSProp and SGD can be seen as approximations of two different regimes of a general Bayesian optimizer.

Small $\eta$

We already supposed that $\eta$ is small, but what happens when $\eta \rightarrow 0$? In this case, the fixed point also converges to zero, and we cannot use this approach directly. Instead, we can use the full update step, which approximates to

Interestingly, this leads exactly to the well-known natural gradient descent algorithm with a learning rate that decreases as $1/t$. It is a bit like an "online" natural gradient optimizer. For reference, the update rule is the same, except that $t$ usually represents the total number of samples in the dataset. The variance decreases as $1/t \v{G}^{-1}$, which is expected as we gather a datapoint at a time, our estimator matches the convergence rate of the Cramer-Rao bound.

Conclusion

In this article, I provided a hopefully more compelling explanation for the success of the Adam optimizer in deep learning compared with regret-based approaches. It changes the interpretation of RMSProp from an "optimization" algorithm searching for a good solution through a loss landscape to a "learning procedure" that incorporates information efficiently by updating a belief using the Bayes rule. From this perspective, Adam is probably close to optimal as a quadratic extension of SGD.

References

Citing this blog post

@misc{plumerault2026adam,
  author = {Plumerault Antoine},
  title = {Why is the Adam Optimizer Working so Well?},
  year = {2026},
}