Journal of Applied Mathematics
Volume 2019 (2019), Article ID 8728196, 6 pages
https://doi.org/10.1155/2019/8728196

Research Article

A New Hybrid Algorithm for Convex Nonlinear Unconstrained Optimization

Eman T. Hamed,¹ Huda I. Ahmed,¹ and Abbas Y. Al-Bayati²

¹Department of Operation Research and Intelligent Techniques, College of Computer Sciences and Mathematics, University of Mosul, Iraq
²University of Telafer, Iraq

Correspondence should be addressed to Huda I. Ahmed; hudaea72@gmail.com

Received 24 December 2018; Revised 16 February 2019; Accepted 13 March 2019; Published 1 April 2019

Academic Editor: Urmila Diwekar

Copyright © 2019 Eman T. Hamed et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract.

In this study, we tend to propose a replacement hybrid algorithmic rule which mixes the search directions like Steepest Descent (SD) and Quasi-Newton (QN). First, we tend to develop a replacement search direction for combined conjugate gradient (CG) and QN strategies. Second, we tend to depict a replacement positive CG methodology that possesses the adequate descent property with sturdy Wolfe line search. We tend to conjointly prove a replacement theorem to make sure global convergence property is underneath some given conditions. Our numerical results show that the new algorithmic rule is powerful as compared to different standard high scale CG strategies.

1. Introduction

The nonlinear CG technique could be a helpful procedure to search out the minimum value of any nonlinear function through exploitation unconstrained nonlinear optimization strategies.

Let us contemplate the subsequent unconstrained minimization problem:

where is a real-valued smooth function. The repetitious formula is given as

where is associate optimum step-size computed by any line search procedure [1]. The search direction is defined as

and denotes , while is a positive scalar.

Well-known established instances of square measure are from Hestenes-Stiefel, Fletcher-Reeves, Polak-Ribière, Liu-Storey, Dai-Yuan, and Dai-Liao (see [2, 3, 3, 4, 4–7], respectively), within the already-existing convergence analysis and implementation of the CG methodology, the weak Wolfe conditions square measure [8]:

If we choose , also the strong Wolfe conditions [8] consist of (4) and

Now, this allows us to review Ibrahim et al. work [9] that could be a work that considers unconstrained minimization problems. Ibrahim et al. recommend a search direction that is outlined as

and is the BFGS updating matrix if step-size is

The scalar is chosen to ensure conjugacy.

Also, another addition is regarded as an equivalent unconstrained minimization problem. Ibrahim et al. [10] recommend another search direction outlined as

Matrix I is the identity and <0.

In addition, Ibrahim et al. [11] propose another search direction that is outlined as

The positive scalar and are the Hestenes-Stiefel parameters.

2. A New Proposed Search Direction

In this section, we advise a replacement search direction as deduced from Ibrahim et al. [9–11]. The new search direction is outlined as

whereas (the approximation matrix of BFGS updating matrix) denotes approximations of Hessian matrix G, and could be a positive constant. In order to drive the value of , we have a tendency to multiply either side of (11) by to induce

Since and (Perry condition [12]), then

In order to see value of , we tend to additionally multiply either side of (11) to induce

As a result of processes of multiplication shown in (13) and (16), we tend to reach the following search directions in (17a), (17b), and (17c). The subsequent new search directions are our new projected algorithmic program:

In the following step, we have a tendency to assume that each search direction got to satisfy the subsequent descent condition (, for all k). Also, there should exist a constant c>0 in order to get

For all , the new direction that is outlined in (18) ought to satisfy the sufficient descent condition. The enough descent conditions are going to be used later to prove our new theorem (see Section 2.2). So to prove our new theorem, we have a tendency to necessarily use the subsequent given assumptions (see Section 2.1).

2.1. Assumptions in [9, 11, 13]

(A1) is twice continuously differentiable.(A2)f is uniformly convex; that is, m and M are positive constants, such that

 for all , and G is the Hessian matrix of f.(A3)The matrix G is Lipschitz continuous at the point ; that is, there exists the positive constant L satisfying

 for all x in a neighborhood of .

2.2. A New Theorem for Proving Sufficient Descent Property

To prove that our new projected algorithm defined in (17a), (17b), and (17c) satisfies sufficiently descent condition, we tend to suppose that assumptions in (Section 2.1) square measure are true. Additionally, the sequence is bounded. Then, sufficient descent condition (18) is true for all .

Proof. When taking (17a), (17b), and (17c) and achieving the descent condition, we can see the following:

We get value , which is bounded away from zero. Therefore (18) is true.

2.3. Lemma in [10, 14]

Suppose that assumptions in Section 2.1 are true. Then, the step-size which is determined by (2) satisfies

when is a positive constant.

2.4. New Theorem for Proving Global Convergence Property

Having demonstrated the important and necessary properties of regression algorithms, we now come to the proof of the necessary property to be present in all numerical optimization algorithms. Let us demonstrate the new algorithm defined in (17a), (17b), and (17c). To achieve a global convergence property, assume that the theory in Section 2.2 and the assumptions in Section 2.1 are correct. Then

Proof. By linking the theory in Section 2.2 and lemma in Section 2.3 we give the following result:

Hence, from our new theorem in Section 2.2, we can define that , and we can therefore simplify (23) as . Thus, the proof is established.

3. A New Form for the Parameter in Conjugate Gradient Method

To obtain an updated version of the conjugate gradient method associated with a new parameter , we compare the standard CG-method specified in (3) with the proposed new algorithm specified in (17a), (17b), and (17c):

Multiplying both sides of (24) by we get

Or

knowing that I is the identity matrix. Moreover,

Then, the new is

and . It should be noted that when using exact line searches assuming that and the matrix is the identity matrix, the new standard will be reduced to HS. This condition must also be met, with (positive constant), and since

after these conditions we know the new parameter is

In conjunction with both parameters

and the search direction is defined as

3.1. Outlines of the New Proposed CG-Algorithm in (29a), (29b), (29c), (29d), and (29e)

By assuming , and by setting the iteration k=1, we get the following steps.

Step 1. Set , if , then stop.

Step 2. Compute by strong Wolfe line search conditions in (4) and in (6).

Step 3. Generate by implementing (29a), (29b), (29c), (29d), and (29e).

Step 4. If [15] is satisfied, then go to Step 1; if not, then continue.

3.2. Assumptions for Proving the Convergence Analysis Property of the New Algorithm in (29a), (29b), (29c), (29d), and (29e)

Let us suppose the following.

(i)The level set

(ii)The condition is satisfied where ; also since f is a uniformly convex function on S, then there exists a constant , such that

From both (19a), (19b), and (31), we can get the following:

3.3. New Theorem for Proving Sufficiently Descent Directions of the New Algorithm in (29a), (29b), (29c), (29d), and (29e)

If we have a tendency to assume that (A3) in Section 2.1 is true and if we assume that conditions (i) and (ii), outlined in (30) and in (31), respectively, are true, then the new proposed search direction defined in (29e) satisfies the sufficient descent condition.

Proof. By using the mathematical induction we demonstrate this new theorem; for initial direction (k=1) we have

which satisfies (18).
We suppose that
And if we multiply both sides of (29a) by , then we can get

since , since :

and since

The above equations ensure that condition (18) is satisfied. Hence, the proof is complete.

3.4. Lemma in [1, 16]

By assuming that assumptions in Section 2.1 are true, by supposing that any CG-method with search direction could be a descent direction provided that the step-size is obtained by the strong Wolfe line search conditions, and if

then

3.5. A New Theorem for Proving the Global Convergence Property of the New Algorithm in (29a), (29b), (29c), (29d), and (29e)

If we suppose that assumptions (i) and (ii) in (30) and in (31), respectively, are true and if we have a tendency to assume that (A3) in Section 2.1 is additionally true, then the search directions outlined in (29e) are descent provided that the step-size is computed using (4) and (6), and then

Proof. since

and since and ,

then from (A3) in part (Section 2.1), from condition (ii) in (31), and from (28) we get

4. Numerical Results and Comparisons

In this work, we have a tendency to compare our new proposed CG-method with some normal classical CG strategies like Hestenes-Stiefel [HS] and Dai-Yuan [DY] by exploitation of fifty unconstrained nonlinear cases; take a look at functions obtained from Andrei [17, 18]. As for the computer program, it was stopped when . In addition, the term , which is defined in (29a), can be computed as . Numerical results for new algorithm in (29a), (29b), (29c), (29d), and (29e) with and are for the total of 50 test problems from the CUTE library. The Sigma plotting software was used to graph the data. We adopt the performance profiles given by Dolan and Moré [19]. Thus, new, HS, and DY strategies are compared in terms of NOI, CPU, and NOF in Figures 1–3. For each method, we plotted the fraction of problems that were solved correctly within a factor of the best time. In the figures, the uppermost curve is the method that solves the most problems within a factor t of the best time. In Figures 1–3, the new method outperforms the HS algorithm and DY method in terms of NOI, CPU, and NOF. If the solution had not converged after 800 seconds, the program was terminated. Generally, convergence was achieved within this time limit; functions for which the time limit was exceeded are denoted by “F” for fail-in.

Figure 1: Performance profiles based on number of iterations.

Figure 2: Performance profiles based on CPU time.

Figure 3: Performance profiles based on function evaluation.

5. Conclusions

At the end of this work we were able to obtain a new direction of research defined in (29a), (29b), (29c), (29d), and (29e). This new trend is a hybrid trend that combines pedigree techniques with Quasi-Newton ones. Through the theories presented in the research, the new trend in (29a), (29b), (29c), (29d), and (29e) proved that it satisfies the requirement of sufficient proportions and ensures the property of global convergence. In addition, we have presented a new scalar () which ensures the sufficient descent directions. Moreover, under some conditions, we have established that the new proposed algorithm is a globally convergent algorithm for uniformly convex functions under the strong Wolfe line search conditions. The numerical results show that when we choose the value of parameters in (13) and in (16), we obtain best numerical results as compared to other similar numerical results.

Data Availability

The data used the support the findings of this study are available from the corresponding author upon request.

Conflicts of Interest

The authors declare that there are no conflicts of interest regarding the publication of this paper.

Acknowledgments

The research is supported by College of Computer Sciences and Mathematics, University of Mosul, Republic of Iraq, under Project no. 8728196.

References

W. Sun and Y.-X. Yuan, Optimization Theory and Methods: Nonlinear Programming, Springer, New York, NY, USA, 2006.

Y. H. Dai and L. Z. Liao, “New conjugacy conditions and related nonlinear conjugate gradients methods,” Applied Mathematical Optimization, vol. 43, pp. 87–101, 2001.

Y. H. Dai and Y. Yuan, “A nonlinear conjugate gradient method with a strong global convergence property,” SIAM Journal on Optimization, vol. 10, no. 1, pp. 177–182, 1999.

R. Fletcher and C. M. Reeves, “Function minimization by conjugate gradients,” The Computer Journal, vol. 7, pp. 149–154, 1964.

M. R. Hestenes and E. Stiefel, “Methods of conjugate gradients for solving linear systems,” Journal of Research of the National Bureau of Standards, vol. 49, pp. 409–436, 1952.

E. Polak and G. Ribière, “Note Sur la convergence de methods de directions Conjugue'e, ESAIM,” Mathematical Modeling and Numerical Analysis, vol. 3, no. 16, pp. 35–43, 1969.

Y. Liu and C. Storey, “Efficient generalized conjugate gradient algorithms, Part 1,” Journal of Optimization Theory and Applications, vol. 69, no. 1, pp. 129–137, 1991.

P. Wolfe, “Convergence conditions for ascent methods(II): some corrections,” SIAM Review, vol. 13, no. 2, pp. 185–188, 1971.

M. A. Ibrahim, M. Mamat, and W. J. Leong, “BFGS method: a new search direction,” Saints Malaysian, vol. 43, pp. 1593–1599, 2014.

M. A. H. Ibrahim, M. Mamat, P. L. Ghazali, and Z. Salleh, “The scaling of hybrid method in solving unconstrained optimization method,” Far East Journal of Mathematical Sciences, vol. 99, no. 7, pp. 983–991, 2016.

M. A. H. Ibrahim, M. Mamat, and A. Z. M. Sofi, “A modified search direction of Broyden family method and its global convergence,” Journal of Engineering and Applied Sciences, vol. 12, no. 17, pp. 4504–4507, 2017.

A. Perry, “A modified conjugate gradient algorithm,” Operations Research, vol. 26, pp. 1073–1078, 1987.

R. Jaafar, M. Mamat, and I. Mohd, “A new scaled hybrid modified BFGS algorithms for unconstrained optimization,” Applied Mathematical Sciences, vol. 7, pp. 263–270, 2013.

J. Nocedal and S. J. Wright, Numerical Optimization, Springer, 2006.

M. J. Powell, “Restart procedure for the conjugate gradient methods,” Mathematics Programming, vol. 12, pp. 241–254, 1977.

H. Yabe and M. Takano, “Global convergence properties of new nonlinear conjugate gradient methods for unconstrained optimization,” Computational Optimization and Application, vol. 28, pp. 203–225, 2004.

N. Andrei, “An unconstrained optimization test functions collection,” Advanced Modeling and Optimization. The Electronic International Journal, vol. 10, no. 1, pp. 147–161, 2008.

N. Andrei, “Open problems in nonlinear conjugate gradient algorithms for unconstrained optimization,” Bulletin of the Malaysian Mathematical Sciences Society. Second Series, vol. 34, no. 2, pp. 319–330, 2011.

E. D. Dolan and J. J. Moré, “Benchmarking optimization software with performance profiles,” Mathematical Programming, vol. 91, no. 2, pp. 201–213, 2002.