2460-0245Journal of the Indonesian Mathematical SocietyJIMS2460-02452086-8952IndoMS10.22342/jims.v32i2.1960Stochastic Influence on Levenberg-Marquardt Method for Nonlinear Least Squares ProblemsChristopherGerendIndonesiagerendc@gmail.comhttps://orcid.org/0000-0003-1789-7187NaiborhuJansonIndonesiajanson@itb.ac.idRizalJosejrizal04@unib.ac.idDepartment of MathematicsBandung Institute of Technologyhttps://ror.org/00apj8t60IndonesiaCorresponding author: Gerend Christopher. Email: gerendc@gmail.com2304202623042026322JUNE1130303202529112025Copyright (c) 2026 Journal of the Indonesian Mathematical Society2026Journal of the Indonesian Mathematical Societyhttps://creativecommons.org/licenses/by-nc-nd/4.0/This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
In this digital era, a lot of data can be collected to be extracted and used for decision making through mathematical models in solving cases in certain domains. This paper will focus on solving nonlinear least squares problems in building an optimal model, namely one that has good accuracy and is efficient. In practice, the model is built using numerical methods. The main objective of this study is to investigate the effect of stochasticity in numerical methods that utilize gradients, namely Levenberg-Marquardt, on accuracy and computational efficiency. In addition, several numerical results from several variants of Stochastic Levenberg-Marquardt sampling and data taken in clusters with K-Means will be compared. The results of this paper are Stochastic Levenberg-Marquardt with 100, 200, 300, 400, and 500 data sample sizes outperformed classical Levenberg-Marquardt since they have very low relative errors to Levenberg-Marquardt and are faster in computational time than Levenberg-Marquardt. Therefore, when solving nonlinear least squares problems with large data sets, the Levenberg-Marquardt method requires only a small number of samples, which suggests that using the Stochastic Levenberg-Marquardt approach is advantageous.
nonlinear least squaresstochastic Levenberg-MarquardtclusteraccuracyK-meanscomputational efficiencyFile created by JATS Editorhttps://jatseditor.comissue-created-year20261. INTRODUCTION
Advancements in mathematics, science, and technology has driven humans to digital and information era. This modern era leads to the exponential growth of creating, collecting, and utilizing data as stated in Kapadiya et al. [1] and Horvitz and Mitchell [2]. As data collection and processing have advanced, the need for efective methods to extract meaningful insights from this data has become increasingly apparent. The development of mathematical models that capable of handling complex dataset is a necessity as well. Hokanson [3] stated that among the various approaches to model data, nonlinear least squares problems have emerged as a critical area of focus, particularly for their applications in data fitting and parameter estimation across diverse fields such as engineering, economics, and machine learning.
Nonlinear Least Squares problems involve finding the best fit curve to a set of data points by minimizing the sum of the squares of the residuals, which usually called the objective function of this problem. This task for finding optimal model that has high accuracy or lower error and has computational eficiency led to the development of numerous of numerical methods [4, 5]. In Nocedal [6] and Yuan [5], Gradient-based numerical methods is one of those methods for solving nonlinear least squares problems.
With the growth of data created, recent advances have explored the integration of stochastic elements into these numerical methods to potentially enhance their performance and its applications [7, 8, 9, 10]. One specific method, Stochastic Gradient Descent that introduce randomness into the optimization process can influence both accuracy and computational eficiency as in Robbins [7] and Kiefer [8]. The impact of such stochastic methods on gradient-based numerical methods remains a topic of ongoing research.
The Levenberg-Marquardt algorithm has been extensively studied for solving nonlinear least squares problems. Early applications include the analysis of EPR spectra (1996) [11] and the development of damping and undamping strategies (1997) [12]. More recent eforts have focused on improving the accuracy and computational eficiency of the algorithm. For example, Yan et al. proposed AdaLM (Adaptive Levenberg-Marquardt), which adapts the step size to enhance performance [13]. Building on this line of work, Hong et al. (2020) introduced a stochastic version of the Levenberg-Marquardt algorithm by using randomly selected subsets of data. [14]. In contrast, our approach introduces a new variation of stochastic sampling: at each iteration, a diferent random subset of the data is selected. Furthermore, this study incorporates K-Means clustering, using cluster centroids as representative data points, which diferentiates our method from previous work.
The main purpose of this research is to investigate the influence of stochastic in numerical methods that utilize gradients, especially for Levenberg-Marquardt method, on accuracy and computational eficiency. In this study, numerical result with data taken in clusters will be compared to Stochastic Levenberg-Marquardt.
2. PROBLEM STATEMENT AND METHODS2.1. Least Squares Problem.
When we have data , the relationship of and can be written in equation (1)
where is a vector consist of model parameters, is a fitting model, and is measurement error on the data ordinate.
For any choice of , residuals can be computed by
The least squares problem is to minimize the sum of the squared residuals by obtaining as the minimizer. Madsen [15] define least squares problem in Definition 2.1
Definition 2.1. [Least Squares Problem] Find a local minimizer
where are given functions, and .
As an objective function, can also be called as loss function or cost function.
2.2. Gradient Descent.
Gradient Descent is an iterative optimization numerical method that uses first order derivative approach with the aim of minimizing the objective function. In Ruder [16], Gradient Descent is a way to minimize objective function parameterized by its model parameters by updating the value of param eters with opposite direction of gradient of objective function. In Tian et al. [17], updating the value of parameters using Gradient Descent can be written as in equation (4)
where denoted as iteration of the simulation. Symbol called as learning rate which usually constant greater than zero that regulate updating steps using gradient. Symbol is all samples used when calculating for every iteration.
Garrigos and Gower [18] showed one way about convergence of Gradient Descent as written in Theorem 2.3
Definition 2.2. Let and 0. \end{document} ]]> is said if it is differentiable and if is
For all
Theorem 2.3.Suppose we want to minimize a diferentiable function and minimizer (argmin) . Assume convex and , 0 \end{document} ]]>. Let be a sequence of iterates generated by Gradient Descent algorithm, with a learning rate satisfying . Then, for all argmin , for all , we have
As the number of data samples increases, the associated computational cost also rises accordingly. Therefore, recent advances introduce randomness or stochastic elements into Gradient Descent. It is done by sampling the data uniformly. In Tian et al. [17] stated that Stochastic Gradient Descent computes gradient only uses one random sample in each iteration. Mini-Batch Stochastic Gradient Descent randomly sampled small-batch data to compute gradient. These two variants save a lot of memory in computation which might lead to speed increased. In this study, Stochastic Gradient Descent term will be used interchangeably for those two variants. With batch data, updating parameters value can be written by
To show one way about the convergence of Stochastic Gradient Descent, Garrigos and Gower [18] depicts the objective function as a sum of functions , where is bounded from below and argmin . They said sum of functions problem as Sum of Convex when is assumed to be convex. They said sum of functions problem as Sum of −smooth when is assumed to be and denoted . Garrigos and Gower [18] showed one way about convergence of Stochastic Gradient Descent as written in Theorem 2.4.
Theorem 2.4.Let the problem hold Sum of and Sum of Convex assumptions. Consider be a sequence of iterates generated by Stochastic Gradient Descent algorithm, with a learning rate satisfying . It follows that
where
with
2.3. Levenberg-Marquardt.
Levenberg-Marquardt (LM) is an iterative optimization numerical method that uses second order derivative approach with the aim of minimizing the objective function. While Gradient Method solely uses first order derivative, this method leverage the second order derivative. As this method may be considered as an extension of Gauss-Newton’s method, the Levenberg-Marquardt also based on solving linearized least squares problems. In Madsen [15], the Levenberg-Marquardt method is based on a linear approximation to the components of in the neighborhood of . For small , the Taylor expansion is
Thus, denoted as
where is Jacobian. Linear approximation for Gradient and Hessian respectively are
Updating the value of using with Levenberg-Marquardt method is done by solving equation (9)
Constant is the damping parameter which influences both the direction and the size of the step [15]. It is stated that the initial value of is related to the size of the elements in by letting
where is chosen by user. Updating value is controlled by gain ratio in equation (11). If is large value, is a good approximation on . If is small, is a poor approximation.
The pseudocode which details the step for updating parameters value using Levenberg-Marquardt are shown in Algorithm 1.
Levenberg-Marquardt
Algorithm 12.4. Stochastic Levenberg-Marquardt.
While Stochastic Gradient Descent randomly sampled data for calculating gradient (first order derivative), in Stochastic Levenberg-Marquardt (SLM), randomly sampled data used for calculating gradient as well as Hessian matrix, specifically for calculating Jacobian matrix (first order and second order derivative) [19]. In Levenberg-Marquardt, Jacobian matrix, gradient, and Hessian matrix are calculated with all data samples, while Stochastic Levenberg-Marquardt calculate those with randomly sampled data. The pseudocode that details the step for updating parameters value using Stochastic Levenberg-Marquardt are shown in Algorithm 2.
2.5. K-Means.
In this study, data will be sampled with centroid-based clustering algorithm, specifically K-Means. Centers from clustering algorithm will represent the data to get the optimized model. K-Means is preferred in this study since the number of centroids (K) could be specified. K-Means algorithm initially proposed by MacQueen [20] in 1967. This method classifies given object to diferent clusters [21].
K-Means algorithm used in this study is K-Means++. Let be data points, let are centers, and let denote the shortest distance from a data point to the closest center which have already chosen, as stated in Arthur and Vassilvitskii [22] the steps for K-Means++ are
Take one center , chosen uniformly at random from .
Take a new center , choosing with probability .
Repeat step 2 until centers have taken altogether.
For each , set the cluster to be the set of points in that are closer to than they are to for all .
For each , set to be the center of mass of all points in .
Repeat step 4 and step 5 until there are no any changes in .
Stochastic Levenberg-Marquardt
Algorithm 23. SIMULATION SETUP3.1. Data and Preprocessing.
Apple’s Stock Price Data (AAPL) from December 12, 1980 to May 16, 2024 was used for building the model. The dataset was taken from Yahoo! Finance with 10,948 rows of data. In this study, degree polynomial model , is built.
In the dataset, there was a diference in and scale. With scale diferent, it caused the sensitivity for model’s parameters. Wan [23] stated that to obtain better convergence using Gradient Descent, the normalization method could help. Thus, in this study, the data were normalized with Min-Max Normalization[24]. This method is chosen since the distribution of and are not normal or close to normal. Min-Max Normalization is shown in equation (12)
Loss function used in this study is Mean Squared Error (MSE) which show in equation (13)
Apart from Levenberg-Marquardt simulation, Stochastic Levenberg-Marquardt with 1, 100, 200, 300, 400, and 500 randomly sampled data were simulated. For Cluster-Levenberg-Marquardt, cluster with 100, 200, 300, 400, and 500 centers were simulated. These numbers were selected after experimenting with random sample far below 100, which had similar result with 1 random sample, and above 500, which had similar result with 500 random samples.
3.2. Parameters Setup.
Initial setups for Levenberg-Marquardt, Stochastic Levenberg-Marquardt and Cluster-Levenberg-Marquardt are
Maximum iteration = 50
In the next section, it will be shown that it does not require higher iteration since the fluctuation of loss function was similar in higher maximum iteration.
As a rule of thumb, small value of is chosen, but the algorithm is not very sensitive by the choice of [15].
Initial .
The initial guess is uniform randomly generated. The initial guess could be chosen by the user or randomly generated.
Computer specifications for running Python simulation program was ASUS Desktop 12th Gen Intel(R) Core(TM) i7-12700, 2100 Mhz 12 Core(s) 20 Logical Processor(s).
4. SIMULATION RESULTS AND DISCUSSION
The loss function comparison of Levenberg-Marquardt, Stochastic Levenberg-Marquardt, and Cluster-Levenberg-Marquardt is displayed in Figure 1. Levenberg-Marquardt has the lowest loss function since it computes the Jacobian using all sample data, as illustrated in Figure 1. It is observed that after completing 50 iterations, the loss function of the stochastic Levenberg-Marquardt with 1 sample model does not approach Levenberg-Marquardt, as shown in Figure 1a, where the loss function is significantly higher than Levenberg-Marquardt. The minimum MSE of SLM 1 sample from 50 iterations is significantly higher, as displayed in Table 1, than the minimum MSE of LM. It is suspected that additional iterations are required for loss function of SLM 1 sample to get closer to LM.
Loss Function Comparison and Relative Error
Method
Minimum MSE
Relative Error (%)
LM
0.001574695911
0
SLM 1
0.239841356826
15130.96333412
SLM 100
0.001575429783
0.046604042483
SLM 200
0.001578011113
0.210529695116
SLM 300
0.001575371382
0.042895308592
SLM 400
0.001576264816
0.099632261320
SLM 500
0.001575898006
0.076338215123
Cl 100
0.001610635152
2.282297240011
Cl 200
0.001606480400
2.018452474183
Cl 300
0.001617227325
2.700928708518
Cl 400
0.001613599200
2.470527099832
Cl 500
0.001612992396
2.431992407551
Next, Figure 1b, Figure 1c, Figure 1d, Figure 1e, and Figure 1f compares loss function for Levenberg-Marquardt, Stochastic Levenberg-Marquardt with 100 samples, and Cluster-Levenberg-Marquardt with 100 centers, and so forth. With Cluster-Levenberg-Marquardt are slightly higher than Levenberg-Marquardt. However, it never approach Levenberg-Marquardt loss function. In Table 1, the relative error of the minimum MSE of Cluster Levenberg-Marquardt to the minimum MSE of Levenberg-Marquardt are below 2.71%. On the other hand, because of the stochastic elements in iterations, in Stochastic Levenberg-Marquardt, there are some iterations has MSE are high, even higher than Cluster-Levenberg-Marquardt. Nevertheless, in certain instances, the MSE of Stochastic Levenberg-Marquardt approaches the MSE of Levenberg-Marquardt. In Table 1, the relative error of the minimum MSE of Stochastic Levenberg-Marquardt to the minimum MSE of Levenberg-Marquardt are below 0.22%. Thus, the accuracy of Stochastic Levenberg-Marquardt is superior than the Cluster Levenberg-Marquardt.
Figure 2 shows the 5th degree polynomial curve with Levenberg-Marquardt, Stochastic Levenberg-Marquardt, and Cluster Levenberg-Marquardt. It is dificult to see any diferences in the curve, except SLM 1 sample, as the relative error are below 2.71%.
Figure 3 shows the computation time comparison among the variants of methods and Table 2 shows total iteration of optimization simulation. For Cluster Levenberg-Marquardt, merely cluster with 100 centers have a lower cumulative computation time compared to Levenberg-Marquardt. In contrast, clusters with 200, 300, 400, and 500 centers have a higher cumulative computation time since K-Means clustering requires a large amount of computation time. On the other hand, despite the fact that the total number of iterations required for the simulation reaches the maximum number, Stochastic Levenberg-Marquardt has lower computation time compared to Levenberg-Marquardt.
Loss Comparison of Levenberg-Marquardt, Stochastic Levenberg-Marquardt, and Cluster-Levenberg-Marquardt
Figure 1.
Curve of 5th Degree Polynomial Model Comparison
Figure 2.
Iteration Comparison
Method
Total Iteration
LM
24
SLM 1
50
SLM 100
50
SLM 200
50
SLM 300
50
SLM 400
50
SLM 500
50
Cl 100
22
Cl 200
25
Cl 300
24
Cl 400
23
Cl 500
24
In general, the computation time of Stochastic Levenberg-Marquardt and Cluster-Levenberg-Marquardt (excluding K-Means computation) are lower than Levenberg-Marquardt as shown in Figure 3. Reducing computational time comes at the cost of accuracy, as these methods typically use smaller sample sizes. Smaller sample sizes contributes in faster computational cost for calculating Jacobian matrix, gradient, and Hessian matrix. Although computational time is reduced, the relative error compared to the Levenberg-Marquardt method remains low as the relative error are below 0.22% for the Stochastic Levenberg-Marquardt approach and below 2.71% for Cluster-Levenberg-Marquardt. With a small relative error compared to the Levenberg-Marquardt method and reduced computation time, this research demonstrates that Stochastic Levenberg-Marquardt outperforms to Levenberg-Marquardt and Cluster-Levenberg-Marquardt in computational eficiency.
Computation Time Comparison
Figure 3.5. CONCLUSION
This research aims to investigate the influence of stochastic in the Levenberg-Marquardt numerical method and sampling using cluster centers on accuracy and computational eficiency. Stochastic Levenberg-Marquardt with 100, 200, 300, 400, and 500 have relative error to Levenberg-Marquardt below 0.22% and have lower computational time in 50 maximum iteration. Cluster Levenberg-Marquardt with 100, 200, 300, 400, and 500 centers have relative error to Levenberg-Marquardt below 2.71% and have higher computational time, except 100 centers. To summarize, Stochastic Levenberg-Marquardt with 100, 200, 300, 400, and 500 outperformed Levenberg-Marquardt and Cluster-Levenberg-Marquardt with very low relative error and reduced computational time, which suggests that using Stochastic Levenberg-Marquardt approach is advantageous. Finally, further research may include a better stopping criterion for Stochastic Levenberg-Marquardt to stop before maximum iteration, applying the methods in bigger datasets, and exploring other clustering methods for better results in Levenberg-Marquardt.
Acknowledgement
This work is supported by Department of Mathematics, Faculty of Mathematics and Natural Sciences, Institut Teknologi Bandung.
REFERENCESA study on largescale applications of big data in modern eraProceedings of the 5th International Conference on Information Management & Machine Intelligence, ACMKapadiyaD.ShekhawatC.SharmaP.202310.1145/3647444.3647880From data to knowledge to action: A global enabler for the 21st centuryHorvitzE.MitchellT.202010.48550/ARXIV.2008.00045Projected nonlinear least squares for exponential fittingSIAM Journal on Scientific Computing396HokansonJ.M.20173107-3128,10.1137/16M1084067Least squares optimization: From theory to practiceRobotics93GrisettiG.GuadagninoT.AloiseI.ColosiM.CorteB.DellaSchlegelD.202051,10.3390/robotics9030051Recent advances in numerical methods for nonlinear equations and nonlinear least squaresNumerical Algebra, Control & Optimization11YuanY.-X.201115-34,10.3934/naco.2011.1.15Numerical OptimizationNocedalJ.WrightS.J.2006Springer10.1007/978-0-387-40065-5A stochastic approximation methodThe Annals of Mathematical Statistics223RobbinsH.MonroS.1951400-407,10.1214/aoms/1177729586Stochastic estimation of the maximum of a regression functionThe Annals of Mathematical Statistics233KieferJ.WolfowitzJ.1952462-466,10.1214/aoms/1177729392A stochastic levenberg– marquardt method using random models with complexity resultsSIAM/ASA Journal on Uncertainty Quantification101BergouE.H.DiouaneY.KungurtsevV.RoyerC.W.2022507-536,10.1137/20M1366253Stochastic levenberg–marquardt neural network implementation for analyzing the convective heat transfer in a wavy finMathematics1110KumarR.S.V.AlsulamiM.D.SarrisI.E.SowmyaG.GamaounF.20232401,10.3390/math11102401Nonlinear-least-squares analysis of slowmotion epr spectra in one and two dimensions using a modified levenberg–marquardt algorithmJournal of Magnetic Resonance Series A1202BudilD.E.LeeS.SaxenaS.FreedJ.H.1996155-189,10.1006/jmra.1996.0113Damping–undamping strategies for the levenberg–marquardt nonlinear leastsquares methodComputers in Physics111LamptonM.1997110-115,10.1063/1Adaptive levenberg–marquardt algorithm: A new optimization strategy for levenberg–marquardt neural networksMathematics917YanZ.ZhongS.LinL.CuiZ.20212176,10.3390/math9172176Stochastic levenberg–marquardt for solving optimization problems on hardware acceleratorsIEEE Conference Proceedings, IEEEHongY.2020Website linkMethods for Non-Linear Least Squares ProblemsMadsenK.NielsenH.B.TinglefO.20042An overview of gradient descent optimization algorithmsRuderS.2016// org/10.48550/ARXIV.1609.04747Recent advances in stochastic gradient descent in deep learningMathematics113TianY.ZhangY.ZhangH.2023682,10.3390/math11030682Handbook of convergence theorems for (stochastic) gradient methodsGarrigosG.GowerR.M.202310.48550/ARXIV.2301.11235Global convergence of a stochastic levenberg–marquardt algorithm based on trust regionJournal of the Operations Research Society of ChinaShaoW.-Y.FanJ.-Y.202410.1007/s40305-023-00529-6Some methods for classification and analysis of multivariate observationsProceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability1MacQueenJ.B.1967281-297,MathResearch on k-means clustering algorithm: An improved k-means clustering algorithm2010 Third International Symposium on Intelligent Information Technology and Security InformaticsNaS.XuminL.YongG.201063-67,IEEE10.1109/IITSI.2010.74k-means++: The advantages of careful seedingProceedings of the Eighteenth Annual ACM-SIAM Symposium on Discrete AlgorithmsArthurD.VassilvitskiiS.20071027-1035,10.5555/1283383.1283494Influence of feature scaling on convergence of gradient iterative algorithmJournal of Physics: Conference Series12133WanX.2019032021,10.1088/1742696/1213/3/032021Impact of data normalization on stock index forecastingInternational Journal of Computer Information Systems and Industrial Management Applications6NayakS.C.MisraB.B.BeheraH.S.2014357-369,