Combining the three-term conjugate gradient method of Yuan and Zhang and the acceleration step length of Andrei with the hyperplane projection method of Solodov and Svaiter, we propose an accelerated conjugate gradient algorithm for solving nonlinear monotone equations in this paper. The presented algorithm has the following properties: (i) All search directions generated by the algorithm satisfy the sufficient descent and trust region properties independent of the line search technique. (ii) A derivative-free search technique is proposed along the direction to obtain the step length
In this paper, the following nonlinear equation is considered:
It is not difficult to show that the solution set of monotone equation (
For
The above equation indicates that
In addition, the hyperplane projection method [
Obviously, the hyperplane
The hyperplane projection method has been proved to possess good theoretical properties and numerical performance for nonlinear monotone equations [
Furthermore, we know that the search directions tend to be poorly scaled in conjugate gradient methods. Consequently, in the line search, more function evaluations must be carried out to obtain an appropriate step length
Inspired by the above discussions, we proposed an accelerated conjugate gradient algorithm that combines the TTPRP method, the acceleration step length, and the hyperplane projection method. The main contributions of the algorithm are as follows: An accelerated conjugate gradient algorithm is introduced for solving nonlinear monotone equations All search directions of the algorithm satisfy the sufficient descent condition All search directions of the algorithm belong to a trust region The global convergence of the presented algorithm is proved The numerical results show that the proposed algorithm is more effective for nonlinear monotone equations The algorithm can be applied to restore an original image from an image damaged by impulse noise
This paper is organized as follows: in the next section, we discuss the ATTPRP algorithm and global convergence analysis. In Section
In this section, we will propose an accelerated algorithm and prove its global convergence. The steps of the given algorithm are as follows.
Step 0: choose any Step 1: stop if Step 2: choose Step 3: if Step 4: let the next iterative value be Step 5: if Step 6: let
The following lemma shows that the search direction
If
In addition, by formula (
The following assumption need to be established in order to study some properties of the ATTPRP algorithm:
The solution set of the problem ( The function
Assumption
In the following paper, if not specifically stated, we always assume that the conditions in Assumption
Let
By line search (
Since
This leads to the ideal inequality (
The following lemma is similar to Lemma
Let the sequence
In particular, the sequence
The above lemma reveals that the distance from the iterative points to the solution set of the problem (
Particularly, we obtain
In the following part, the global convergence and the strong global convergence properties of the ATTPRP algorithm will be proven.
Let
We will prove this theorem by contradiction. Supposing that the equation (
According to Lemma
This contradicts with formula (
The following theorem indicates the strong global convergence of the ATTPRP algorithm, which is similar to Theorem
Let
Theorem
In this section, the numerical experiments will be divided into two parts for illustration. The first subsection involves normal nonlinear equations, and the second subsection describes image restoration problems. All tests in this section are coded in MATLAB R2017a, run on a PC with Intel (R) Core (TM) i5-4460 3.20 GHz, 8.00 GB of SDRAM memory, and Windows 7 operating system.
In this subsection, we perform some numerical experiments to show the effectiveness of the ATTPRP algorithm. Some test problems and their relevant initial points are listed as follows:
Exponential
Initial guess:
Exponential
Initial guess:
Singular function:
Initial guess:
Logarithmic function:
Initial guess:
Broyden tridiagonal function:
Initial guess:
Trigexp function:
Initial guess:
Strictly convex
Initial guess:
Variable dimensioned function:
Initial guess:
Tridiagonal system:
Initial guess:
Five-diagonal system:
Initial guess:
Extended Freudenstein and Roth function (
For
Initial guess:
Brent problem:
Initial guess:
To test the numerical performances of the ATTPRP algorithm, we also perform the experiments with the LS algorithm and the TTPRP algorithm. The columns of Tables NO: the serial number of the problem Dim: the variable NI: the number of iterations NF: the number of iterations of the function value CPU: the calculation time in seconds GN: the final function norm evaluations when the program is stopped Initialization: the parameters are chosen as Stop rule: when the condition
From Tables
Test results of the ATTPRP algorithm.
No. | Dim | ATTPRP algorithm | ||
NI/NF | CPU | GN | ||
1 | 3000 | 123/124 | 0.608404 | 9.97 |
9000 | 88/89 | 0.733205 | 9.86 | |
30000 | 57/58 | 1.092007 | 9.96 | |
90000 | 38/39 | 1.310408 | 9.85 | |
2 | 3000 | 28/514 | 0.686404 | 9.50 |
9000 | 18/386 | 1.372809 | 9.22 | |
30000 | 15/371 | 3.432022 | 9.83 | |
90000 | 10/276 | 3.946825 | 9.95 | |
3 | 3000 | 14016/20709 | 107.172687 | 1.00 |
9000 | 15769/23986 | 331.158923 | 9.99 | |
30000 | 15609/27514 | 574.09928 | 1.00 | |
90000 | 17136/35208 | 1503.022835 | 1.00 | |
4 | 3000 | 63/558 | 1.029607 | 5.24 |
9000 | 106/1096 | 5.241634 | 8.39 | |
30000 | 195/2337 | 24.258155 | 1.20 | |
90000 | 336/4521 | 88.031364 | 2.21 | |
5 | 3000 | 90/615 | 0.780005 | 8.52 |
9000 | 96/759 | 2.464816 | 7.76 | |
30000 | 130/1194 | 11.154072 | 8.62 | |
90000 | 176/1984 | 29.250188 | 8.82 | |
6 | 3000 | 91/1223 | 1.918812 | 8.09 |
9000 | 136/2093 | 8.845257 | 9.86 | |
30000 | 221/3905 | 38.282645 | 9.97 | |
90000 | 379/7177 | 125.814807 | 9.33 | |
7 | 3000 | 48/335 | 0.468003 | 9.66 |
9000 | 71/636 | 2.480416 | 8.57 | |
30000 | 122/1339 | 13.135284 | 9.31 | |
90000 | 203/2562 | 43.63348 | 8.53 | |
8 | 3000 | 1/2 | 0.000001 | 0.00 |
9000 | 1/2 | 0.0468 | 0.00 | |
30000 | 1/2 | 0.000001 | 0.00 | |
90000 | 1/2 | 0.0156 | 0.00 | |
9 | 3000 | 6014/88059 | 100.885847 | 9.93 |
9000 | 6437/103883 | 312.345202 | 9.91 | |
30000 | 7345/137142 | 1209.803355 | 9.96 | |
90000 | 8950/196012 | 2763.386114 | 9.99 | |
10 | 3000 | 1811/19670 | 25.693365 | 9.99 |
9000 | 1947/22554 | 76.36249 | 9.82 | |
30000 | 2202/28399 | 261.644877 | 9.95 | |
90000 | 2638/38968 | 577.983705 | 1.00 | |
11 | 3000 | 351/4205 | 5.382034 | 9.78 |
9000 | 409/5194 | 16.645307 | 9.44 | |
30000 | 510/7139 | 64.475213 | 9.64 | |
90000 | 683/10789 | 161.351834 | 9.52 | |
12 | 3000 | 184/188 | 0.156001 | 9.98 |
9000 | 184/188 | 0.577204 | 9.98 | |
30000 | 184/188 | 1.107607 | 9.98 | |
90000 | 184/188 | 3.07322 | 9.98 |
Test results of the TTPRP algorithm.
No. | Dim | TTPRP algorithm | ||
NI/NF | CPU | GN | ||
1 | 3000 | 129/130 | 0.436803 | 9.97 |
9000 | 89/90 | 0.873606 | 9.96 | |
30000 | 59/60 | 1.232408 | 9.96 | |
90000 | 41/42 | 1.435209 | 9.73 | |
2 | 3000 | 46/1055 | 1.435209 | 9.99 |
9000 | 10/267 | 0.951606 | 6.76 | |
30000 | 9/278 | 2.667617 | 9.83 | |
90000 | 8/279 | 4.040426 | 9.00 | |
3 | 3000 | 17401/18908 | 109.481502 | 9.98 |
9000 | 19999/23227 | 355.089476 | 1.37 | |
30000 | 19431/26598 | 612.241525 | 9.99 | |
90000 | 19999/34409 | 1542.678289 | 2.56 | |
4 | 3000 | 70/662 | 1.123207 | 3.49 |
9000 | 113/1326 | 6.099639 | 2.14 | |
30000 | 196/2809 | 28.766584 | 6.47 | |
90000 | 334/5473 | 101.884253 | 3.11 | |
5 | 3000 | 54/464 | 0.514803 | 3.38 |
9000 | 72/711 | 2.152814 | 3.75 | |
30000 | 100/1225 | 10.99807 | 7.38 | |
90000 | 160/2256 | 31.403001 | 5.61 | |
6 | 3000 | 84/1417 | 2.246414 | 6.94 |
9000 | 127/2473 | 10.311666 | 9.41 | |
30000 | 211/4669 | 45.349491 | 8.60 | |
90000 | 346/8524 | 141.773709 | 8.53 | |
7 | 3000 | 45/366 | 0.546003 | 2.67 |
9000 | 68/729 | 2.808018 | 1.23 | |
30000 | 117/1558 | 14.617294 | 8.34 | |
90000 | 195/3039 | 48.219909 | 7.08 | |
8 | 3000 | 1/2 | 0.0624 | 0.00 |
9000 | 1/2 | 0.000001 | 0.00 | |
30000 | 1/2 | 0.0624 | 0.00 | |
90000 | 1/2 | 0.0624 | 0.00 | |
9 | 3000 | 6521/123863 | 139.277693 | 9.81 |
9000 | 6827/141140 | 425.679929 | 9.97 | |
30000 | 7711/182289 | 1559.916399 | 9.99 | |
90000 | 9107/251132 | 3460.398582 | 9.85 | |
10 | 3000 | 4904/65319 | 77.750898 | 1.00 |
9000 | 5271/71854 | 229.820673 | 9.73 | |
30000 | 5280/75948 | 689.150018 | 9.97 | |
90000 | 5655/88105 | 1269.302137 | 9.99 | |
12 | 3000 | 300/4529 | 4.898431 | 9.88 |
9000 | 387/6163 | 17.846514 | 9.61 | |
30000 | 472/8397 | 72.836867 | 9.83 | |
90000 | 620/12490 | 168.652681 | 9.27 | |
13 | 3000 | 193/198 | 0.124801 | 9.99 |
9000 | 193/198 | 0.405603 | 9.99 | |
30000 | 193/198 | 0.826805 | 9.99 | |
90000 | 193/198 | 2.589617 | 9.99 |
Test results of the LS algorithm.
No. | Dim | LS algorithm | ||
NI/NF | CPU | GN | ||
1 | 3000 | 174/175 | 0.982806 | 9.96 |
9000 | 94/95 | 0.670804 | 9.88 | |
30000 | 60/61 | 1.107607 | 9.93 | |
90000 | 41/42 | 1.248008 | 9.97 | |
2 | 3000 | 61/1405 | 1.762811 | 9.85 |
9000 | 34/916 | 3.05762 | 9.98 | |
30000 | 15/470 | 3.946825 | 9.46 | |
90000 | 13/456 | 6.864044 | 9.12 | |
3 | 3000 | 19999/21506 | 125.721206 | 1.43 |
9000 | 19999/23222 | 351.267452 | 1.44 | |
30000 | 19999/27153 | 634.003664 | 2.24 | |
90000 | 19999/34389 | 1582.942147 | 1.76 | |
4 | 3000 | 75/666 | 1.185608 | 7.17 |
9000 | 118/1331 | 6.24004 | 6.87 | |
30000 | 202/2813 | 30.186194 | 7.82 | |
90000 | 338/5467 | 96.159016 | 7.65 | |
5 | 3000 | 136/1145 | 1.404009 | 8.63 |
9000 | 150/1358 | 4.305628 | 9.81 | |
30000 | 171/1806 | 16.801308 | 5.69 | |
90000 | 234/2865 | 42.08907 | 9.58 | |
6 | 3000 | 109/1695 | 2.698817 | 7.38 |
9000 | 147/2694 | 11.263272 | 8.98 | |
30000 | 235/4934 | 47.502305 | 8.67 | |
90000 | 364/8724 | 147.062143 | 7.57 | |
7 | 3000 | 69/400 | 0.530403 | 6.22 |
9000 | 94/765 | 2.948419 | 3.29 | |
30000 | 142/1590 | 15.085297 | 1.79 | |
90000 | 217/3070 | 50.450723 | 4.55 | |
8 | 3000 | 1/2 | 0.0624 | 0.00 |
9000 | 1/2 | 0.000001 | 0.00 | |
30000 | 1/2 | 0.0468 | 0.00 | |
90000 | 1/2 | 0.0156 | 0.00 | |
9 | 3000 | 5643/107964 | 117.484353 | 9.84 |
9000 | 6131/128776 | 373.934397 | 9.86 | |
30000 | 6997/169540 | 1461.323767 | 9.99 | |
90000 | 8449/239554 | 3276.130201 | 9.88 | |
10 | 3000 | 4171/55825 | 65.47362 | 9.93 |
9000 | 4213/58137 | 183.113974 | 1.00 | |
30000 | 4616/67344 | 608.747102 | 9.88 | |
90000 | 4965/79166 | 1150.008172 | 9.92 | |
11 | 3000 | 3484/49132 | 52.073134 | 9.92 |
9000 | 371/5921 | 16.504906 | 9.85 | |
30000 | 3625/52579 | 454.727315 | 9.93 | |
90000 | 3802/57077 | 801.049535 | 9.87 | |
12 | 3000 | 194/199 | 0.109201 | 9.96 |
9000 | 194/199 | 0.405603 | 9.96 | |
30000 | 194/199 | 0.998406 | 9.96 | |
90000 | 194/199 | 2.776818 | 9.96 |
Performance profiles of the methods (NI).
Performance profiles of the methods (NF).
Performance profiles of the methods (CPU).
The purpose of this subsection is to recover the original image from an image damaged by impulse noise. It has important practical significance in optimization fields. The selection of parameters is similar to that in the above subsection. The stop condition is
Restoration of the Cameraman, Barbara, and Man images by using the ATTPRP algorithm and TTPRP algorithm. From left to right: a noisy image with 30% salt-and-pepper noise and the restorations obtained with the ATTPRP algorithm and the TTPRP algorithm by minimizing
Restoration of the Cameraman, Barbara, and Man images by using the ATTPRP algorithm and TTPRP algorithm. From left to right: a noisy image with 50% salt-and-pepper noise and the restorations obtained with the ATTPRP algorithm and TTPRP algorithm by minimizing
Restoration of the Cameraman, Barbara, and Man images by using the ATTPRP algorithm and TTPRP algorithm. From left to right: a noisy image with 70% salt-and-pepper noise and the restorations obtained with the ATTPRP algorithm and TTPRP algorithm by minimizing
CPU times of the ATTPRP algorithm and TTPRP algorithm in seconds.
30% noise | Cameraman | Barbara | Man | Total |
ATTPRP algorithm | 2.184 | 4.789 | 20.514 | 27.487 |
TTPRP algorithm | 2.23 | 5.179 | 20.748 | 28.157 |
50% noise | Cameraman | Barbara | Man | Total |
ATTPRP algorithm | 3.276 | 9.142 | 35.475 | 47.893 |
TTPRP algorithm | 3.307 | 9.204 | 35.677 | 48.188 |
70% noise | Cameraman | Barbara | Man | Total |
ATTPRP algorithm | 3.619 | 13.073 | 58.812 | 75.504 |
TTPRP algorithm | 3.978 | 13.4 | 59.249 | 76.627 |
From Figures
In this paper, an accelerated conjugate gradient algorithm that combines the TTPRP method, the acceleration step length, and the hyperplane projection technique is proposed. All search directions
For future research, we have some ideas as follows: (i) If the acceleration system is introduced into the quasi-Newton method, does it have some good properties? (ii) Can the acceleration system be introduced into the trust region method to solve unconstrained optimization problems and nonlinear equations? (iii) Can the proposed algorithm be applied to machine learning?
The data used to support the findings of this study are included within the article.
The authors declare that they have no conflicts of interest.
This work was supported by the National Natural Science Foundation of China (Grant No. 11661009), the High Level Innovation Teams and Excellent Scholars Program in Guangxi Institutions of Higher Education (Grant No. (2019)52), and the Guangxi Natural Science Key Fund (No. 2017GXNSFDA198046).