A Modified Nonlinear Conjugate Gradient Method with the Armijo Line Search and Its Application

In this article, a modiﬁed Polak-Ribi`ere-Polyak (PRP) conjugate gradient method is proposed for image restoration. The presented method can generate suﬃcient descent directions without any line search conditions. Under some mild conditions, this method is globally convergent with the Armijo line search. Moreover, the linear convergence rate of the modiﬁed PRP method is established. The experimental results of unconstrained optimization, image restoration, and compressive sensing show that the proposed method is promising and competitive with other conjugate gradient methods.


Introduction
Consider the following unconstrained optimization problem: where f(x): R n ⟶ R is a continuously differentiable function. e iteration formula is determined by where α k is stepsize obtained by some kind of line search and d k is search direction. It is well known that there are many methods ( [1][2][3][4] etc.) for solving optimization problem (1), where the conjugate gradient method is a powerful method because of its simplicity, and this method can avoid the computation and storage of some matrices associated with the Hessian of objective functions; then its memory requirements are very low. Large-scale calculations are usually required in the process of image processing; therefore, the conjugate gradient algorithm has excellent application prospects in this aspect. e search direction d k of the conjugate gradient methods is given by where β k ∈ R is a scalar and g k � g(x k ) is the gradient of objective function f(x) at the point x k . e choice of β k determines different conjugate gradient methods. Classical conjugate gradient methods include PRP conjugate gradient method [5,6], HS conjugate gradient method [7], LS conjugate gradient method [8], DY conjugate gradient method [9], FR conjugate gradient method [10], and CD conjugate gradient method [11]. e parameters β k of these methods are specified as follows: where y k− 1 � g k − g k− 1 , and ‖·‖ stands the Euclidean norm. ere has been much research on convergence properties of these methods ( [12][13][14][15] etc.) and applications ( [16][17][18][19][20][21][22][23] etc.).
In this paper, the focus is mainly on the PRP method. e PRP method is generally believed to have the best numerical performance in classical conjugate gradient methods, but there is much room for discussion in terms of convergence. When the exact line search was used, the global convergence of the PRP conjugate gradient method has been proved by Polak and Ribière [5] for convex objective functions. However, Powell [24] proposed a counter example that the PRP conjugate gradient method may fail for nonconvex functions even if the exact line search is used. Gilbert and Nocedal [14] showed a modified PRP conjugate gradient method that the modified PRP method is globally convergent if β PRP k is restricted to be not less than zero and α k is determined by a line search step satisfying the sufficient descent condition: in addition to the following weak Wolfe-Powell conditions, i.e., where 0 < σ < φ < 1. Meanwhile, Gilbert and Nocedal [14] pointed out that even if the objective function is guaranteed to be uniformly convex, β k may be negative. With the strong Wolfe-Powell line search, Dai [25] gave an example showing that even though the objective function is uniformly convex, the PRP method cannot guarantee that the search direction is always the descent direction. rough the above observations and [14,24,[26][27][28], the sufficient descent condition (5) and the condition β k ≥ 0 play key roles in establishing the global convergence of the conjugate gradient methods. However, in the case where Armijo line search or Wolfe line search is used, the descent property of d k determined by (3) is in general not guaranteed.
In order to obtain the generated sufficient descent direction, Hager and Zhang [29] showed a new conjugate gradient method (CG-C) obtained by modifying the HS method, which generates sufficient descent directions at each step, without relying on any line search. e scalar β k in CG-C method is determined by where and μ > 0 is a constant. With the weak Wolfe-Powell line search, Hager and Zhang [29] established a global convergence result for (7) when the objective function f(x) is general nonlinear function.
Along this line, Yu and Guan [30,31] extended the CG-C method to the PRP, LS, DY, FR, and CD methods. e parameter β k in the modified PRP method (DPRP) is as follows: where η > (1/4) is a constant. e performance of the above methods is better than other conjugate gradient in practice.
In addition, because Armijo line search is very simple, compared with Wolfe line search, it is more widely used in image restoration and compressed sensing problems. erefore, it is necessary to explore the performance of the above methods under Armijo line search. However, the above methods cannot obtain global convergence with Armijo line search, i.e., to find a stepsize where 0 < δ, ρ < 1 are given constants. In order to overcome this defect, scholars have done a lot of related work ( [32][33][34] etc). In order to overcome the defect of PRP, HS and LS conjugate gradient methods cannot globally converge with the Armijo line search, Li and Qu [33] proposed a series of modified conjugate gradient methods based on (3) and (4) as follows: where z k− 1 is a scalar to be specified. e parameter z k− 1 is max t‖d k− 1 ‖, ‖g k− 1 ‖ 2 in the modified PRP method (APRP). And the results of global convergence are established when the Armijo line search is used. Motivated by the idea of [31,33] and taking into account the excellent numerical performance, we propose a new modified PRP method. e parameter β k is computed as where z k− 1 � max t‖d k-1 ‖, ‖g k− 1 ‖ 2 , t > 0 and η > (1/4) are two constants. And the search direction d k is determined by the classical two-term conjugate gradient method (3).
In the next section, the proposed algorithm is stated. In Section 3, the properties and the convergent results of the new method are given. is method enjoys linear convergence under suitable conditions in Section 4. Numerical results and conclusion are presented in Section 5 and in Section 6, respectively.

Algorithm
is section will give algorithm of the new modified PRP method in association with the Armijo line search technique (10) for (1). e algorithm is stated as follows.
Remark 1. Without any line search method, the NPRP method can get sufficient descent properties. Lemma 1. Let x k and d k be generated by the NPRP method.

Properties and Convergence Analysis
e following few suitable assumptions are often utilized in global convergence analysis for conjugate gradient algorithms.
(ii) Function f: R n ⟶ R is continuously differentiable and its gradient is Lipschitz continuous on an open convex set Ω 1 containing Ω, i.e., there exists a constant L > 0 such that It follows directly from Assumption 1 that there is a positive constant A, such that Lemma 2. If Assumptions 1 holds, x k , α k , d k , and g k be generated by Algorithm 1. ere exists a constant B > 1 such that the following inequality holds for all k, then Proof. When k � 0, from (3), we can obtain then (21) holds. When k ≥ 1, from (3) and (20), we have Mathematical Problems in Engineering Together with (3) and the above formula implies letting us (21) Proof. Case (i) α k � 1. From (14) and (26), we get en we need to satisfy From Lemma 2, we have Case (ii) α k < 1. By the Armijo line search condition, ρ − 1 α k does not satisfy inequality (10). is means By the mean-value theorem and inequality (19), there is a p k ∈ (0, 1) such as x k + p k ρ − 1 α k d k ∈ Ω 1 and Observing the last inequality and (30), we have letting us (26) is satisfied.
□ Lemma 4. Let the sequence x k , d k and g k be generated by Algorithm 1, and suppose that Assumption 1 holds, then, Proof. We have from (10) and Assumption 1 that Substituting (26) into the above formula, we have where C and δ are two constants. erefore, (34) Together with (21), the above formula implies where B � 1 + (2LA/t) + (4η(LA) 2 /t 2 ). From the above inequality, we can obtain lim k⟶∞ ‖g k ‖ � 0.

Linear Convergence Rate
When discussing the linear convergence rate for conjugate gradient methods, the following assumptions are often established on the basis of Assumption 1.

Assumption 2
i.e., there exist positive constants M ≥ m > 0 such that Theorem 2. Suppose Assumption 2 holds. Let x * be the unique solution of problem (1) and the sequence x k be generated by the NPRP method with the Armijo line search. en there are constants a > 0 and r ∈ (0, 1) such that for all k, then that is, x k converges to x * at least R-linearly.

Numerical Experiments
In this section, we report some numerical results of Algorithm 1 and compare the performance of some methods previously detailed in this paper. e following methods were compared: Step 0: Choose an initial point x 0 ∈ R n , constants δ, ρ ∈ (0, 1). Compute. g 0 � ∇f(x 0 ). Set k � 0.
Step 2: Compute the system of equations (3) to get d k , where β k is obtained by (13).
Step 3: Determine a step size α k by the Armijo line search (10).
All programs are written in MATLAB R2019a and run on a PC with an AMD Ryzen 5 3550 H with Radeon Vega Mobile Gfx CPU @2.10 GHz, 16 GB (14.9 GB available) of RAM and the Windows 10 operating system.

Tested Problems.
A number of 74 unconstrained optimization test problems are described in Table 1 [36,37].

Termination
Rule. If g k < 10 − 6 or if the number of iterations exceeds 10 3 .

Symbol Representation.
No: the test problem number, CPUTime: the CPU time in seconds, NI: the number of iterations, NFG: the total number of function and gradient evaluations. Figures 1-3 show the performance profiles of CPU Time, NI, and NFG of the three methods.

Image Description.
From Table 2 and Figures 1-3, we can see the performance of NPRP method and APRP method is significantly better than that of DPRP method. e reason is that the Armijo line search is more convenient than the Wolfe line search to obtain stepsize α k . Meanwhile, we can get that the performance of NPRP method is slightly better than that of APRP method.

Image Restoration Problems.
is subsection is done to recover the original image from an image corrupted by impulse noise.

Contrast Algorithm.
is experiment tests two algorithms, NPRP and APRP.

Compressive Sensing Problems.
e purpose of this section is to accurately recover the image from a few random projections by compressive sensing (CG), based on the

Termination Rule.
If the square root of the sum of the diagonal elements of g T k g k is less than 10 − 3 or if the number of iterations exceeds 500.

Symbol
Representation. PSNR: peak signal to noise ratio. e greater the PSNR, the better the effect. Figure 4: From left to right: 25% noise, NPRP method, APRP method. Figure 5: From left to right: 50% noise, NPRP method, APRP method.  Table 4 and Figure 7 show us at least two conclusions: (i) NPRP method and APRP method are successful for restoring these images with suitable PSNR; (ii) NPRP method is promising and competitive to APRP method for compressive sensing problems.
From the above numerical experiment results, we can see that the performance of the NPRP method is more competitive than the APRP method and the DPRP method. e reasons are as follows. (i) Hager and Zhang pointed out in the survey paper [39] that the common molecule g T k y k− 1 of the PRP, HS, and LS methods possess a built-in restart feature that addresses the jamming problem: the factor y k− 1 � g k − g k− 1 in the numerator of β k tends to zero when the step s k− 1 � x k − x k− 1 is too small. In addition, β k becomes smaller, and the new search direction d k is closer to the steepest descent direction g k . Meanwhile, NPRP method is similar in form to CG-C and PRP methods and therefore inherits their good performance. (ii) e Armijo line search is more convenient than the Wolfe line search to obtain stepsize α k .

Conclusion
In this paper, a modified PRP conjugate gradient method was proposed to solve image restoration and compression sensing problems, based on the well-known CG-C method [29]. With the Armijo line search, the global convergence and liner convergence rate of the algorithm is established under some suitable conditions. e sufficient descent property of the algorithm has been proved without the use of any line search method. e numerical results indicate that the algorithm is effective and competitive for solving unconstrained optimization problems, image restoration problems, and compressive sensing problems.

Data Availability
e data used to support the findings of this study are included within the article.

Conflicts of Interest
ere are no potential conflicts of interest.