An Accelerated Conjugate Gradient Algorithm for Solving Nonlinear Monotone Equations and Image Restoration Problems

Combining the three-term conjugate gradient method of Yuan and Zhang and the acceleration step length of Andrei with the hyperplane projection method of Solodov and Svaiter, we propose an accelerated conjugate gradient algorithm for solving nonlinear monotone equations in this paper. The presented algorithm has the following properties: (i) All search directions generated by the algorithm satisfy the suﬃcient descent and trust region properties independent of the line search technique. (ii) A derivative-free search techniqueisproposedalongthedirectionto obtainthesteplength α k .(iii)If ϕ k � − α k ( h k − h ( w k )) T d k > 0,thenan accelerationscheme is used to modify the step length in a multiplicative manner and create a point. (iv) If the point satisﬁes the given condition, then it is the next point; otherwise, the hyperplane projection technique is used to obtain the next point. (v) The global convergence of the proposed algorithm is established under some suitable conditions. Numerical comparisons with other conjugate gradient algorithms show that the accelerated computing scheme is more competitive. In addition, the presented algorithm can also be applied to image


Introduction
In this paper, the following nonlinear equation is considered: where h: R n ⟶ R n is continuous and monotone, and h(x) satisfies It is not difficult to show that the solution set of monotone equation (1), unless empty, is convex. is problem has many significant applications in applied mathematics, economics, and engineering. For example, the economic equilibrium problem [1] can be transformed into problem (1). Generally, an iteration formula generates the next iteration point by where α k is the step length and d k is a search direction, which are two important factors for solving nonlinear equations.
Some derivative-free line search techniques [2][3][4][5] were proposed to search for step length α k . Li and Li [6] presented a derivative-free line search to find α k � max ρs i : i � 0, 1, . . . , } such that where ρ, λ > 0 and s ∈ (0, 1). ‖.‖ represents the Euclidean norm. e line search technique (4) is different from other existing derivative-free line search techniques because it does not use a merit function. If d k satisfies h(x k ) T d k < 0, the inequality (4) holds for all sufficiently small α k > 0. As a result, α k can be obtained by some backtracking processes. For d k , it is well known that the Newton methods [7,8], quasi-Newton methods [9][10][11][12][13], spectral gradient methods [14,15], and conjugate gradient methods [16,17] can deal with large-scale nonlinear equations. For solving large-scale optimization problems, the conjugate gradient methods are quite effective since they only calculate and store the gradient value of the objective function. Many scholars have applied conjugate gradient theory to solve nonlinear monotone equations and have achieved good results [17][18][19]. Classical conjugate gradient methods include HS method [20], FR method [21], PRP method [22,23], LS method [24], CD method [25], and DY method [26]. In particular, the PRP method, as one of the most effective methods, generates a small step near a minimum point; then, the subsequence generated for the search direction d k will automatically approach the negative gradient direction, which avoids continuously generating a small step. However, the global convergence of the PRP method is not established under inexact line search techniques for general functions. Many scholars have performed continuous research and have reached satisfactory conclusions. Zhang [27] proposed the MPRP method, where d k is designed as follows: It is easy to obtain from (5) that e above equation indicates that d k is a descent direction of h at x k . If the exact line search is used, then we have θ k � 0. Consequently, formula (5) is inferred to be the standard PRP method. Under some mild conditions, the MPRP method is globally convergent under the Armijo-type line search, but global convergence cannot be established under the weak Wolfe-Powell line search. e main reason is that the MPRP method does not satisfy the trust region property. Inspired by the above discussions, Yuan and Zhang [28] proposed a three-term PRP (TTPRP) method in which d k is defined by where κ > 0 is a constant. It is worth noting that the denominator ‖h k− 1 ‖ 2 in the formula of the MPRP method is adjusted to max κ‖d k− 1 ‖‖y k ‖, ‖h k− 1 ‖ 2 in the formula of the TTPRP method. e TTPRP method automatically maintains the trust region property. Its global convergence is also established under certain conditions. Numerical results show that the TTPRP method is effective for large-scale nonlinear monotone equations. In addition, the hyperplane projection method [17,[29][30][31][32] is the most effective method for solving largescale nonlinear monotone equations. e concept of projection was first proposed by Goldstein [33] for convex programming in a Hilbert space. Furthermore, Solodov and Svaiter [34] proposed the hyperplane projection method for solving optimization problems. e specific process of the hyperplane projection method is as follows: let x k be the current iteration point, and obtain a point w k � x k + α k d k along a certain line search direction d k such that h(w k ) T (x k − w k ) > 0. Due to the monotonicity of h(x), for a certain point x that satisfies h(x) � 0, it can be deduced that Obviously, the hyperplane H k � x ∈ R n | 〈h(w k ), x − w k 〉 � 0} strictly separates the current iteration point x k from the solution set of the equation (1). e point x k is projected onto the hyperplane H k to obtain the next iteration point x k+1 , i.e., e hyperplane projection method has been proved to possess good theoretical properties and numerical performance for nonlinear monotone equations [6,34].
Furthermore, we know that the search directions tend to be poorly scaled in conjugate gradient methods. Consequently, in the line search, more function evaluations must be carried out to obtain an appropriate step length α k . Andrei [35] presented an acceleration scheme that modifies the step length α k in a multiplicative manner to improve the reduction of the function values along the iterations. e step length is defined as follows: where A numerical comparison with some conjugate gradient algorithms shows that the computational scheme is effective.
Inspired by the above discussions, we proposed an accelerated conjugate gradient algorithm that combines the TTPRP method, the acceleration step length, and the hyperplane projection method. e main contributions of the algorithm are as follows: An accelerated conjugate gradient algorithm is introduced for solving nonlinear monotone equations All search directions of the algorithm satisfy the sufficient descent condition All search directions of the algorithm belong to a trust region e global convergence of the presented algorithm is proved e numerical results show that the proposed algorithm is more effective for nonlinear monotone equations e algorithm can be applied to restore an original image from an image damaged by impulse noise is paper is organized as follows: in the next section, we discuss the ATTPRP algorithm and global convergence analysis. In Section 3, we report the preliminary numerical experiments to show that the algorithm is efficient for nonlinear monotone equations and applicable to image restoration problems. In Section 4, the conclusion regarding the proposed algorithm is given.

Accelerated Algorithm and Convergence Analysis
In this section, we will propose an accelerated algorithm and prove its global convergence. e steps of the given algorithm are as follows.
Step 4: let the next iterative value be w k � x k + α k d k .
e following lemma shows that the search direction d k designed by using formula (7) has not only the sufficient descent property but also the trust region property independent of the line search. Lemma 1. d k is defined by using formula (7); then, we obtain Proof. If k � 0, formulas (7) and (12) are obviously true. If k ≥ 1, we obtain from formula (7) that In addition, by formula (7), we get ‖d k ‖ ≥ ‖h k ‖ and where max κ‖d k− 1 ‖‖y k ‖, ‖h k− 1 ‖ 2 ≥ κ‖d k ‖‖y k ‖. en, the proof is completed. e following assumption need to be established in order to study some properties of the ATTPRP algorithm: In the following paper, if not specifically stated, we always assume that the conditions in Assumption 1 hold.

Lemma 2.
Let x k and w k be generated by using the ATTPRP algorithm. e step length α k generated by the ATTPRP algorithm satisfies where Proof. By line search (4), assuming α k ≠ ρ, let α k ′ ≜ α k s − 1 , by the definition α k ′ , does not satisfy the line search (4). at is, Since h(x) is Lipschitz continuous and by using formula (11), we have namely, is leads to the ideal inequality (17). e proof is completed.
e following lemma is similar to Lemma 1 in the study of Solodov and Svaiter [34], which also holds for the Mathematical Problems in Engineering ATTPRP algorithm. erefore, we only state it as follows but omit its proof. □ Lemma 3. Let the sequence x k be generated by using the ATTPRP algorithm. Suppose that x is a solution of problem (1) with h(x) � 0. We obtain In particular, the sequence x k is bounded, and Remark 2. e above lemma reveals that the distance from the iterative points to the solution set of the problem (1) decreases along iterations. Otherwise, for any k, it is followed from formulas (9) and (4) that Particularly, we obtain In the following part, the global convergence and the strong global convergence properties of the ATTPRP algorithm will be proven. Theorem 1. Let x k be generated by using the ATTPRP algorithm. en, we have Proof. We will prove this theorem by contradiction. Supposing that the equation (25) does not hold, there exists a constant η > 0 such that ‖h k ‖ ≥ η holds for all k ≥ 0. From formula (12), we have According to Lemma 3 and equation (24), the sequences x k and w k ′ are bounded. By formulas (7) and (16), for all k ≥ 1, we obtain where max κ‖d k− 1 ‖‖y k ‖, ‖h k− 1 ‖ 2 ≥ ‖h k− 1 ‖ 2 . Since ‖x k+1 − x k ‖} converges to zero, the last inequality shows that ‖d k ‖ is bounded. By formulas (12) and (16), we obtain where M � (1 + 2/κ)φ. us, from the formulas (16) and (17), we obtain that is contradicts with formula (24). Consequently, the proof is completed. e following theorem indicates the strong global convergence of the ATTPRP algorithm, which is similar to eorem 1 in [6]. We also give a specific proof for convenience of understanding. □ Theorem 2. Let x k be generated by using the ATTPRP algorithm.
en, the whole sequence x k converges to a solution of the problem (1).
Proof. eorem 1 shows that there exists a subsequence of x k converging to a solution x of the problem (1). On the other side, it follows from Lemma 3 that the sequence ‖x k+1 − x‖ converges. erefore, the whole sequence x k converges to x.

Numerical Experiments
In this section, the numerical experiments will be divided into two parts for illustration. e first subsection involves normal nonlinear equations, and the second subsection describes image restoration problems. All tests in this section are coded in MATLAB R2017a, run on a PC with Intel (R) Core (TM) i5-4460 3.20 GHz, 8.00 GB of SDRAM memory, and Windows 7 operating system.

Normal Nonlinear Equations.
In this subsection, we perform some numerical experiments to show the effectiveness of the ATTPRP algorithm. Some test problems and their relevant initial points are listed as follows: Function 1. Exponential Function 1: Initial guess: x 0 � (n/n − 1, n/n − 1, . . . , n/n − 1) t .

Function 12.
Brent problem:   Tables 1-3, it is obvious that the three methods can successfully solve most of the test problems with NI<20000. However, for Function 3 with 9000 and 90000 variables, the   [36] proposed a drawing tool that can obtain the performance profiles of methods. erefore, using the drawing tool, we obtain Figures 1-3, which are related to the NI, NF, and CPU in Tables 1-3. In Figure 1, the ATTPRP algorithm solves all test problems at approximately τ � 1.8, while the LS algorithm solves 80% of the test problems at approximately τ � 1.88, and the TTPRP algorithm solves 92% at approximately τ � 19. us, we can obtain the result that the ATTPRP algorithm performs slightly better than the other two algorithms. When τ � 1.45 in Figure 2, the presented algorithm solves all test problems, the TTPRP algorithm solves 87% of all test problems, and the LS algorithm only      solves approximately 70% of the test problems. us, it is not difficult to see that the ATTPRP algorithm is more competitive than the other two methods. In Figure 3, the curve of the ATTPRP algorithm is above those of the TTPRP and LS algorithms, which indicates that the proposed algorithm is more robust than the other two algorithms in terms of the CPU. In summary, the enhancement of the presented method is noticeable.

Image Restoration
Problems. e purpose of this subsection is to recover the original image from an image damaged by impulse noise. It has important practical significance in optimization fields. e selection of parameters is similar to that in the above subsection. e stop condition is |‖h k+1 ‖ − ‖h k ‖|/‖h k ‖ < 10 − 3 or ‖x k+1 − x k ‖/‖x k ‖ < 10 − 3 . For the experiments, Cameraman (256 × 256), Barbara (512 × 512), and Man (1024 × 1024) are chosen as the test images. We also perform experiments to compare the ATTPRP algorithm with the TTPRP algorithm, where the step length α k is generated by Step 2 and Step 3 in the ATTPRP algorithm. More detailed performance results are shown in Figures 4-6. It is not difficult to see that both the ATTPRP and TTPRP algorithms are successful in the image restoration of the three images. e expenditure of the CPU time is listed in Table 4 to compare the ATTPRP algorithm with the TTPRP algorithm.
From Figures 1-4, we can obviously note that both algorithms can perfectly restore a noisy image with 30%, 50%, and 70% salt-and-pepper noise. In addition, the results in Table 4 show that the ATTPRP algorithm and the TTPRP algorithm are both successful in restoring these images with an approximate CPU time. e presented algorithm is slightly competitive with the TTPRP algorithm for 30% noise problems, 50% noise problems, and 70% noise problems.

Conclusions
In this paper, an accelerated conjugate gradient algorithm that combines the TTPRP method, the acceleration step length, and the hyperplane projection technique is proposed. All search directions d k generated by using the algorithm automatically have sufficient descent and trust region properties. e global convergence property of the proposed algorithm is established under suitable conditions. e numerical results show that the proposed algorithm is effective. e image restoration problems also demonstrate that the proposed algorithm is successful For future research, we have some ideas as follows: (i) If the acceleration system is introduced into the quasi-Newton method, does it have some good properties? (ii) Can the acceleration system be introduced into the trust region method to solve unconstrained optimization problems and nonlinear equations? (iii) Can the proposed algorithm be applied to machine learning?

Data Availability
e data used to support the findings of this study are included within the article.

Conflicts of Interest
e authors declare that they have no conflicts of interest.