Publication record · 18.cifr/2014.kingma.adam-optimizer

Adam: A Method for Stochastic Optimization

v1.0.0

Diederik P. Kingma (University of Amsterdam), Jimmy Ba (University of Toronto)

RAI18.cifr/2014.kingma.adam-optimizer

ICLR 2015· 2014· doi:10.48550/arXiv.1412.6980

We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments. The method is straightforward to implement, is computationally efficient, has little memory requirements, is invariant to diagonal rescaling of the gradients, and is well suited for problems that are large in terms of data and/or parameters.

stochastic optimizationadaptive learning ratesgradient descentdeep learningonline learning

✦ Research context

What this agent contributes to the literature.

Problem solved

Standard SGD requires careful global learning-rate tuning and performs poorly on sparse or noisy gradients. Researchers training large models needed a computationally cheap, memory-efficient optimizer robust to hyperparameter choices. Adam addresses this by maintaining per-parameter adaptive rates that require little tuning in practice.

Novelty

Adam combines adaptive learning rates with bias-corrected moment estimates, solving the cold-start problem of RMSProp and AdaGrad. The bias correction terms (1-beta1^t) and (1-beta2^t) ensure reliable gradient scaling from the very first iteration. AdaMax, a variant using the infinity norm, is also introduced as an extension.

Related research

Computing related research...

Canvas contract1-in / 1-out · unpacked into problem, hyperparams legacy ports

Sample data

Loading sample data...

Total calls

This month

Citations

Last called

—

Image digest

sha256:d0ab3df27862f4649b279b7f10a588c70a3f069f8bbffa4c95151c1069663bbd

Invoke command

python main.py

Inputs

input:application/json

Outputs

output:application/json

Citation

Loading DOI…

Invoke

CPU compute only

How to get GPU access: Your university, lab, or company can become a CIFR institutional member. Members get GPU-accelerated runs for all their researchers. Contact us

Recent invocations(0)

No invocations yet — be the first to call this agent.