Adaptive Gradient Convergence under PL and Non-Uniform Smoothness

Tajinder Singh

Adaptive Gradient Convergence under PL and Non-Uniform Smoothness

Author(s)	Mr. Tajinder Singh
Country	India
Abstract	The Adam optimizer dominates modern deep learning practice, yet classical convergence theory has long struggled to justify this preference. Standard analyses anchored in uniform L-smoothness and bounded-gradient assumptions yield only the canonical stochastic rate \mathbb{E}\min\below{t\in\left[1,T\right]}\parallel\nabla f\left(\theta_t\right)\parallel\le\mathcal{O}\left(T^{-1/4}\right), a guarantee already saturated by Stochastic Gradient Descent with Momentum (SGDM) . The empirical separation between Adam and (S)GD on Transformer-class workloads therefore remains unexplained by such bounds. This review synthesizes two parallel lines of recent theory that, taken together, repair the gap. The first replaces the global Lipschitz hypothesis with the \left(L_0,L_1\right)-smooth model of , under which Adam admits an upper rate \mathcal{O}\left(\left(L_0+L_1\right)\sigma_0^2\left(f\left(\theta_1\right)-f^\ast\right)/\varepsilon^4\right) that strictly dominates a constructed SGDM lower bound. The second invokes the Polyak-Łojasiewicz (PL) inequality, \sfrac{1}{2}\parallel\nabla f\left(\theta\right)\parallel^2\geq\mu\left(f\left(\theta\right)-f^\ast\right), to upgrade sublinear \mathcal{O}\left(1/\sqrt T\right) guarantees to linear convergence in expectation for both AdaGrad and Adam . A third strand, grounded in high-probability tail analysis, attributes Adam’s advantage to second-moment normalization, which produces a confidence dependence of \mathcal{O}\left(\delta^{-1/2}T^{-1/2}\right), sharper than the \mathcal{O}\left(\delta^{-1}T^{-1/2}\right) floor for vanilla SGD . We argue that these threads constitute a coherent unification: non-uniform smoothness explains why Adam is adaptive, the PL geometry explains how fast it converges on practically relevant landscapes, and tail-control explains why its trajectories concentrate sharply. Open problems regarding parameter-agnostic schedules and per-coordinate dimension dependence are catalogued at the close.
Keywords	Adaptive Gradient Methods, Adam Optimizer, Non-Uniform Smoothness, Polyak-Łojasiewicz Inequality, High-Probability Bounds, Convergence Analysis
Field	Mathematics
Published In	Volume 8, Issue 2, March-April 2026
Published On	2026-03-07

View / Download PDF File

About IJFMR Fees & Payment Current Issue Publication Archive	Submit Research Paper Track Submission Status Publication Guidelines Publication Ethics Peer Review & Plagiarism	Join as a Reviewer Editors & Reviewers Reviewer Referral Program Get Reviewer Membership Certi.	Website/Journal Policies Usage Policy Content Policies Privacy Policy

Contact Us		+91-9687-828-838	editor@ijfmr.com

International Journal For Multidisciplinary Research

A Widely Indexed Open Access Peer Reviewed Multidisciplinary Bi-monthly Scholarly International Journal

Adaptive Gradient Convergence under PL and Non-Uniform Smoothness

Share this