
Source: Almost Surely Math
What is Newton’s Method
When training a deep learning model, the goal is often to minimize a loss function over many parameters. Optimization methods, which determine how parameters are updated during training, are used to achieve this goal. In practice, most deep learning systems rely on first-order methods, such as stochastic gradient descent or Adam. However, second-order methods like Newton’s Method also provide good solutions in theory, as they can provide faster and more accurate convergence under ideal conditions.
Newton’s Method approaches optimization problems by using first and second order derivatives. Rather than relying solely on the gradient, it also incorporates the Hessian matrix, which captures the local curvature of the loss landscape. While gradient descent moves parameters in the direction of steepest descent, Newton’s Method uses curvature information to choose both the direction and magnitude of each update step.
Understanding Newton’s Method
To better understand how Newton’s Method works, consider the task of finding the minimum of a complex function. By using Newton’s Method, you can first calculate the function’s first derivation, which represents the slope or steepness of the function at any given point. You can then compute the second derivative, the rate at which the slope is changing. Next, you can choose a point on the function that is believed to be the true minimum and use this derivative information to construct a local quadratic approximation, known as a Taylor approximation, which resembles the original function in a neighborhood around that point. The minimum of this Taylor approximation can then be computed analytically and used as the next estimate of the true minimum. By repeating this process, Newton’s Method iteratively moves closer to the true minimum of the original function.

Source: AIML.com Research
Newton’s Method in equation form:
where
- $x_{k+1}$ is the updated estimate of the minimizer after applying one Newton step, expected to be closer to the true minimum than $x_k$
- $f(x)$ is the loss function being minimized
- $f'(x_k)$ is the first derivative of $f(x)$ evaluated at $x_k$, representing the gradient of the function at the current point and indicating the direction of the steepest increase
- $f”(x_k)$ is the second derivative (Hessian) of $f(x)$ evaluated at $x_k$ which measures the curvature of the function at the current point
Why Newton’s Method Provides Good Solutions
In machine learning, the optimization goal is to find a parameter vector that minimizes a loss function. Newton’s Method achieves this by leveraging a second-order Taylor expansion of the loss function around the current parameter estimate. This provides several advantages over first-order optimization methods.
First, Newton’s Method benefits from curvature-aware updates. The Hessian encodes how sharply or smoothly the loss function curves in different directions. As a result, the method automatically scales step sizes. It takes smaller steps in directions with steep curvature and larger steps in flatter directions. This avoids the inefficient zig-zag behavior often seen in gradient descent when the loss surface is ill-conditioned.
Second, Newton’s Method exhibits quadratic convergence near an optimum. Once the parameters are close to a local minimum, the error decreases extremely rapidly. This means that very few iterations are required to reach a highly accurate solution, especially compared to first-order methods, which typically converge linearly.
Third, Newton’s Method provides more direct steps toward the optimum. Because it models the loss locally as a quadratic function, it can move directly toward the minimum of that approximation rather than relying on incremental progress. This makes it particularly effective for convex problems such as logistic regression and classical statistical models.
Due to these properties, Newton’s Method often finds high-quality solutions efficiently and serves as an idealized benchmark for optimization algorithms in machine learning.

Source: AIML.com Research
Why Newton’s Method is Difficult to Use in Practice
Despite its strong theoretical guarantees, Newton’s Method is rarely used directly in deep learning due to several major practical challenges.
The most significant limitation is computational cost arising from the curse of dimensionality. For a model with $n$ parameters, the Hessian matrix has size $n$ x $n$. Modern deep neural networks can have millions or even billions of parameters, making it infeasible to compute, store, or invert the Hessian. The cost of inverting such a matrix grows cubically with the number of parameters, making real-world use impractical.
Another major issue is non-convexity. While Newton’s Method works well for convex functions, learning loss functions are highly non-convex. When this occurs, Newton’s Method may produce updates that move toward saddle points or even increase the loss. This instability is particularly problematic far from the optimum, where the quadratic approximation may be inaccurate.
Additionally, Newton’s Method is not well suited for stochastic optimization. Deep learning typically uses mini-batches of data to estimate gradients, which introduces noise. Estimating second-order derivatives under stochastic settings leads to highly noisy Hessian approximations, undermining the reliability of Newton-style updates.
Finally, Newton’s Method can be unstable during early training. Although it converges quickly near an optimum, its behavior is far from a good solution and can be erratic. It often requires additional mechanisms such as damping or trust-region constraints.
Solutions and Adaptations
Because of these challenges, researchers have developed approximate second-order methods inspired by Newton’s Method that are more practical for deep learning.
Quasi-Newton methods, such as BFGS (Broyden–Fletcher–Goldfarb–Shanno) and L-BFGS (Limited-memory Broyden–Fletcher–Goldfarb–Shanno), avoid computing the Hessian explicitly. Instead, they approximate the inverse Hessian using information from past gradients.
Diagonal Hessian approximations simplify computation by retaining only the diagonal elements of the Hessian. This significantly reduces memory and computational requirements and is used in methods such as AdaHessian, which adapts learning rates using approximate curvature information.
Block-diagonal and layer-wise approximations exploit neural network structure by approximating curvature separately for different layers. Methods such as K-BFGS and Natural Gradient approaches fall into this category, balancing curvature awareness with scalability.
More specialized methods, such as CureNewton, introduce cubic regularization to handle degenerate or indefinite Hessians, particularly in tasks like neural network unlearning, where precise parameter adjustments are required.
Conclusion
Newton’s Method is a foundational second-order optimization algorithm that uses both gradient and curvature information to achieve fast and accurate convergence. In theory, it provides superior solutions compared to first-order methods, especially near optimal points, and eliminates many issues related to learning-rate tuning and poor conditioning.
However, the enormous computational cost of computing and inverting the Hessian, combined with the non-convex and stochastic nature of deep learning, makes Newton’s Method impractical in its pure form. As a result, modern deep learning relies on scalable first-order methods while borrowing ideas from Newton’s Method through approximations such as quasi-Newton techniques and curvature-aware optimizers.
Video Explanations:
- This video “Visually Explained: Newton’s Method in Optimization” by Visually Explained provides a visual overview of Newton’s Method, the intuition behind it, and some of its pros and cons. (Runtime: 11 mins)
- This video “Newton’s Method for optimization” by OptiML PSE explains Newton’s Method. Starting at 13:00, the video discusses the performance of Newton’s Method, as well as its advantages and disadvantages, compared to first-order methods such as Gradient Descent. (Runtime: 4 mins)
Related Articles:

