A New Modified Efficient Levenberg–Marquardt Method for Solving Systems of Nonlinear Equations

For systems of nonlinear equations, a modified efficient Levenberg–Marquardt method with new LM parameters was developed by Amini et al. (2018).*e convergence of the method was proved under the local error bound condition. In order to enhance this method, using nonmonotone technique, we propose a new Levenberg–Marquardt parameter in this paper.*e convergence of the new Levenberg–Marquardt method is shown to be at least superlinear, and numerical experiments show that the new Levenberg–Marquardt algorithm can solve systems of nonlinear equations effectively.


Introduction
Consider the system of nonlinear equations where the function F(x): R n ⟶ R m is continuously differentiable. In this paper, we assume that the solution set of (1) (denoted by X * ) is nonempty, with ‖ · ‖ referring to the 2norm. Newton method is an important method to solve system (1) in [1]. At each iteration, it uses the trial step where F(x k ) � F k and J k � F ′ (x k ) is Jacobian matrix. When J(x) is Lipschitz continuous and nonsingular, then the convergence of this method is quadratic at the solution. However, trial step d N k may not exist and J k is singular or near singular. Newton method may not be well defined. To overcome this difficulty, Levenberg-Marquardt (LM) method was created by Levenberg [2] and Marquardt [3] which uses the trial step d LM k at each iteration, where and λ k is a nonnegative constant. By introducing a nonnegative parameter λ k , LM method overcomes the problem that J k is singular or near singular; furthermore, excessive step size ‖d k ‖ is avoided. In this case, where λ k � 0 and Jacobian matrix J k is nonsingular, the LM method is reduced to Newton method. e efficiency of the LM method is affected by the parameter λ k . For example, let λ k � ‖F(x k )‖ 2 , under the local error bound condition, the LM method is shown to have quadratic convergence by Yamashita and Fukushima in [4]. However, when the sequence x k is far away from the set X * , ‖F k ‖ may be very big which may lead to large λ k . It will result in a smaller LM step size, further reducing the efficiency of the algorithm. In [5], Fan used λ k � μ k ‖F(x k )‖ δ , δ ∈ (1,2], where μ k is updated with trust region technology in each iteration, the LM method also has quadratic convergence under some suitable conditions, and λ k � μ k ‖F(x k )‖ δ can alleviate the effect of the initial point being far away from the set X * .
To avoid this trouble, Amini used λ k � μ k ‖F k ‖/(1 + ‖F k ‖) in [6]; when the sequence x k is far from the solution set and ‖F k ‖ is very large, λ k is close to μ k , which effectively controls the range of λ k . Umar proposed some new LM parameters λ k � (‖J k ‖/‖J k ‖ 2 ), λ k � (‖J T k J k ‖/ ‖J T k J k ‖ 2 ) in [7]. Wang used λ k � η k ‖J T k F k ‖ α with η k updated by trust region techniques from iteration to iteration in [8].
Ma introduced ‖J T k F k ‖ into LM method and used a new LM parameter λ k � θ‖F k ‖ + (1 − θ)‖J T k F k ‖ in [9], where 0 ≤ θ ≤ 1. It is noticeable that λ k is a convex combination of ‖F k ‖ and ‖J T k F k ‖, and the quadratic convergence of this method is proved. ere are numerous other various LM methods to solve (1); interested readers are referred to [10][11][12] for related work. In order to discuss the range of parameter λ k , inspired by [6,8,9], in this paper, we choose a new LM parameter as follows: where λ k is a convex combination of and μ k is updated with trust region technology in each iteration. Now, we set as the merit function for (1). We define the actual reduction and the predicted reduction of ϕ(x) at the kth iteration as follows: where d k is computed by (3). e following ratio is Grippo applied the nonmonotone line search technique to Newton's method in [13]; some authors have extended the nonmonotone techniques to trust region algorithm and proposed a lot of effective nonmonotone trust region methods in [14,15]. And Amini proposed nonmonotone line search technique for the LM method in [6]. Numerical experiments show that the algorithm with the nonmonotone technique is more efficient than the algorithm without the nonmonotone technique. Inspired by these theories, we apply a nonmonotone strategy to LM method in this paper. Let us replace actual reduction (6) with the following actual reduction: where n(k) � min N 0 , k , and N 0 is a positive integer constant. Obviously, by this change, ‖F k+1 ‖ will be compared with the max 0≤j≤n(k) ‖F k− j ‖ in each iteration, further leading to affect the ratio. e ratio after the change is It can be used to decide whether the trial step is accepted and update the trust region parameter μ k . e paper is organized as follows. In Section 2, we present a new algorithm and then prove the global convergence of the new algorithm under some conditions. In Section 3, under the local error bound condition, the convergence of the new Levenberg-Marquardt method is shown to be at least superlinear. In Section 4, the new algorithm is an effective algorithm, which is demonstrated by numerical results. At last, we give some conclusions in Section 5.

The Efficient Algorithm and Global Convergence
In this section, firstly, we present the new efficient LM algorithm and then prove the global convergence of the new algorithm.
When sequence x k is close to the solution, the steps may be too large, so we require in the new algorithm, where m is a positive constant, and this is implemented by Step 5.

Lemma 1. For all k ∈ N, we have
Proof. is proof is directly derived from the important theory given by Powell in [16].
From literature [6], the following lemma can be obtained. □ Lemma 2. Suppose the sequence x k be generated by Algorithm 1, then the sequence F l(k) converges.
Next we present some of the assumptions needed in the following content.
(b) J(x) is Lipschitz continuous; i.e., there exists a positive constant L 1 such that Lemma 3. If Assumption 1 holds, then we have Proof. e proof of (17) can be found in [17]. So, we only prove (16). Using mean value theorem, there exists z ∈ [x, y] that makes and hence, According to the last equation, we can obtain So (16) Proof. Assume that the theorem is incorrect; then, there exist a positive constant ε 0 and a constant k 0 ∈ N that makes Firstly, we prove that Since d k is accepted by the algorithm, we have is, along with (17), (22), and Lemma 1, for all k ≥ k 0 , that means Replacing k with l(k) − 1, for all sufficiently large k, there is which together with the last inequality yields Using Assumption 1, the last equality implies that Let l(k) � l(k + N 0 + 2). Using induction, for all j ≥ 1, we can show that For j � 1, we can have from (31) that (29) is true. Assuming that (29) is true for given j, we show that (29) holds for given j + 1. Let k be large enough such that l(k) − (j + 1) > 0. Substituting k with l(k) − j − 1 and using (24), we obtain Similarly, we can deduce that erefore (31) holds. Along with Assumption 1 we imply that Similarly, for any given j ≥ 1, we have lim k→∞ On the one hand, for any k, we have Using (31) and the fact that With Assumption 1, we conclude

Local Convergence
Assumption 2 (a) F(x) is continuously differentiable, and ‖F(x)‖ provides a local error bound on subset N(x * , b) for problem (1), where Lemma 4. Suppose that Assumption 2 is true. en, for all sufficiently large k, we have the following.
(1) ere exists a positive constant M > m that makes Proof.
e proof process of (1) is the same as that of Lemma 3.2 in [6], so we are not going to prove it here and only give the proof of (2).
We can obtain from Definition 1 and (45) and (46) that us we obtain en we show the following inequalities: If ‖F k ‖ ≤ 1, then the following holds: and if ‖F k ‖ > 1, then 4 Mathematical Problems in Engineering rough the above two inequalities, we have Similarly, if ‖J T k F k ‖ ≤ 1, then the following holds Hence, we obtain From Algorithm 1, (52), and (58), we have where Proof. If we set φ k (d) � ‖F k + J k d‖ 2 + λ k ‖d‖ 2 , then we have from (3) that d k is a minimizer of φ k (d), so it follows from Algorithm 1, (12), (45), (46), and ‖F(x k )‖ � 0 that So there is Lemma 6. Suppose that Assumption 2 is true, for all sufficiently large k. So, we have Proof. Firstly, we deal with these two equations: On the other hand, We conclude where Mathematical Problems in Engineering 5 Without loss of generality, for all x ∈ N(x * , b) ∩ X * , suppose that rank(J(x)) � r, and we prove the local convergence of Algorithm 1 by singular value decomposition (SVD) of J(x). where And assume the SVD of J(x) are as follows: Since J(x) is Lipschitz continuous, by the theory of matrix perturbation [18], we have So there is Since x k converges to the set X * , then we have L 1 ‖(x k − x k )‖ ≤ (σ r /2) hold for all sufficiently large k. So combined with (71), there is Proof. e proof process is similar to Lemma 7 in [9], so we omit it here. Proof. Using (3) and (69), we obtain From (47), (71), Definition 1, and Lemma 5, we have From (44), (46), (74), Definition 1, and Lemma 5, we conclude that On the other hand, it is obvious that It follows from (75) and Lemma 5 that

Numerical Experiments
In Section 5, we compare the performance of Algorithm 1 with Algorithm 2.1 (writing Algorithm 2.1 as AELM) in [6] through some numerical experiments. e test function F(x) is improved by the method in [19]. e form is as follows:
By numerical experiments, we find the numerical results of Algorithm 1 are the same as the numerical results of AELM in some functions. So we only list the results of other experiments in the following tables. Further, we adopt the efficiency index defined as EI in [21] to compare the performance of algorithm AELM and Algorithm 1. e results of the four experiments with rankJ(x * ) � n − 1 are shown in Tables 1 and 2, and the results of the four experiments with rankJ(x * ) � n − 2 are shown in Tables 3 and 4, respectively. We use six starting points ±100x 0 , ±10x 0 , and ±x 0 for each test problem, where x 0 is suggested in [20].
(i) NF stands for the quantity of function calculations (ii) NJ stands for the quantity of Jacobian calculations (iii) '− ' indicates that the iteration number is more than 10 4 (iv) E.I. � ρ 1/NF , where ρ is the convergence order of algorithm.
It is shown in Table 1 that when θ � 0.5 and δ � 2, the effect of Algorithm 1 is obviously better than that of AELM. Algorithm 1 wins 40.5% of the numerical results while AELM wins 2.38%, and 57.1% of the two algorithms have the same results. e advantage of Algorithm 1 is not obvious when θ � 0.5 and δ � 1. Algorithm 1 can win 19% of the numerical results while AELM win 7.14%, and 73.8% of two algorithms have the same result. Table 3 shows that when θ � 0.5 and δ � 1, Algorithm 1 and AELM have the best experimental results. Algorithm 1 win 23.3% of the numerical results, and 76.6% of the two algorithms has the same result. e advantage of Algorithm 1 is not obvious when θ � 0.5 and δ � 2. Algorithm 1 wins 40% of the numerical results while AELM wins 20%, and 40% of the two algorithms has the same results.
Further, we adopt the EI and let θ � 0.5 in the experiment. Tables 2 and 4 show that when δ � 1, the experimental data EI of AELM and Algorithm 1 are similar, but when δ � 1, the EI of Algorithm 1 is obviously larger than that of AELM. In addition, in terms of the experimental time, except when ranking (F′(x * ) � n − 2) and δ � 1, the execution time of Algorithm 1 is longer than that of AELM. In other cases, the execution time of Algorithm 1 is close to or less than that of AELM.
In general, it is shown that for most test problems, Algorithm 1 performs better than AELM. So it can indicate that Algorithm 1 is more efficient than AELM to solve systems of nonlinear equations.

Conclusion
In this paper, we propose a new LM algorithm by modifying the LM parameter for systems of nonlinear equations.
rough numerical experiments, we find the calculation amounts of Algorithm 1 smaller than AELM in the case where θ and δ take some suitable value, which shows the effectiveness of the new Algorithm 1. Under some conditions, the global convergence of the new LM method is proved, and the local convergence of the new LM method is shown to be at least superlinear. Numerical results show that the new algorithm is efficient.

Data Availability
No data were used to support the findings of this study.

Conflicts of Interest
e authors declare that they have no conflicts of interest.