^{1}

^{1, 2}

^{1}

^{2}

The minimization of a quadratic function within an ellipsoidal trust region is an important subproblem for many nonlinear programming algorithms. When the number of variables is large, one of the most widely used strategies is to project the original problem into a small dimensional subspace. In this paper, we introduce an algorithm for solving nonlinear least squares problems. This algorithm is based on constructing a basis for the Krylov subspace in conjunction with a model trust region technique to choose the step. The computational step on the small dimensional subspace lies inside the trust region. The Krylov subspace is terminated such that the termination condition allows the gradient to be decreased on it. A convergence theory of this algorithm is presented. It is shown that this algorithm is globally convergent.

Nonlinear
least squares (NLS) problems are unconstrained optimization problems with
special structures. These problems arise in many aspects such as the solution
of overdetermined systems of nonlinear equations, some scientific experiments,
pattern recognition, and maximum likelihood estimation. For more details about
these problems [

There are two general types of algorithms for solving
NLS problem (

The presented algorithm is a Newton-Krylov type
algorithm. It requires a fixed-size limited storage proportional to the size of
the problem and relies only upon matrix vector product. It is based on the
implicitly restarted Arnoldi method (IRAM) to construct a basis for the Krylov
subspace and to reduce the Jacobian into a Hessenberg matrix in conjunction
with a trust region strategy to control the step on that subspace [

Trust region methods for unconstrained minimization
are blessed with both strong theoretical convergence properties and a good
accurate results in practice. The trial computational step in these methods is
to find an approximate minimizer of some model of the true objective function
within a trust region for which a suitable norm of the correction lies inside a
given bound. This restriction is known as the trust region constraint, and the
bound on the norm is its radius. The radius is adjusted so that successive model
problems minimized the true objective function within the trust region [

The trust region subproblem is the problem of finding

There are two different approaches to solve (

Several authors have studied inexact Newton's methods
for solving NLS problems [

This contribution is organized as follows. In Section

The subspace technique plays an important role in
solving the NLS problem (

The first order necessary conditions of (

The model trust region algorithm generates a sequence
of points

The solution

Let

The main result which is used to prove that the
sequence of gradients tends to zero for modified Newton methods is

The algorithm
we will discuss here requires that

the algorithm should be well defined for a sufficiently general class of functions and it is globally convergent;

the algorithm
should be invariant under linear affine scalings of the variables, that is, if
we replace

the algorithm should provide a decrease that is at least as large as a given multiple of the minimum decrease that would be provided by a quadratic search along the steepest descent direction;

the algorithm
should give as good a decrease of the quadratic model as a direction of the
negative gradient when the Hessian,

The following describes a full iteration of a truncated Newton type method. Some of the previous characteristics will be obvious and the other ones will be proved in the next section.

Step 0 (initialization).

Given

Choose

Step 1 (construction a basis for the Krylov
subspace).

Choose

For

Form

Compute the residual norm

If

Step 2 (compute the trial step).

Construct the local model,

Compute the solution

Step 3 (test the step and update

Evaluate

If

begin

Comment: Do not accept the step “

Else if

Comment: Accept the step and keep the previous trust region radius the same

Else

Comment: Accept the step and increase the trust region radius

end if

Step 4.

Set

In this section, we discuss the important role of the restarting mechanism to control the possibility of the failure of nonsingularity of the Hessian matrix, and introduce the assumptions under which we prove the global convergence.

Let the sequence of the iterates generated by
Algorithm

for all

for all

An immediate consequence of the global assumptions is
that the Hessian matrix

Assumption G3 that

To handle the problem of singular

We have discussed the possibility of stagnation in the linear IRA method which results in a break down in the nonlinear iteration. Sufficient conditions under which stagnation of the linear iteration never occurs are

we have to
ensure that the steepest descent direction belongs to the subspace,

there is no
difficulty, if one required the Hessian matrix,

One of the main restrictions of most of the
Newton-Krylov schemes is that the subspace onto which a given Newton step is
projected must solve the Newton equations with a certain accuracy which is
monitored by the termination condition (assumption G4). This condition is
enough to essentially guarantee convergence of the trust region algorithm.
Practically, the main difficulty is that one does not know in advance if the
subspace chosen for projection will be good enough to guarantee this condition.
Thus,

In this section, we are going to establish some
convergence properties which are possessed by Algorithm

Let the global assumptions hold. Then
for any

Suppose

The following lemma shows that the termination norm
assumption G4 implies that the cosine of the angle between the gradient and the
Krylov subspace is bounded below.

Let the global assumptions hold. Then

Suppose

The following Lemma establishes that Algorithm

Let the
global assumptions hold and let

Since,

The following two facts will be used in the remainder of the proof.

By Taylor's
theorem for any

For any
sequence

The next result establishes that every limit point of
the sequence

Let the global assumptions hold and Algorithm

Since

Let

The following lemma proves that under the global
assumptions, if each member of the sequence of iterates generated by Algorithm

Let the global assumptions hold,

Let

Since

Find

We claim that

The following lemma establishes the rate of
convergence of the sequence of iterates generated by Algorithm

Let the assumptions of Lemma

To show that the convergence rate is super linear, we
will show eventually that

To show that eventually the truncated-Newton step is
always shorter than the trust region radius, we need a particular lower bound
on

The next result shows that under the global
assumptions every limit point of the sequence

Let the global assumptions hold, and
assume

Suppose to the contrary that

In this paper, we have shown that the implicitly restarted Arnoldi method can be combined with the Newton iteration and trust region strategy to obtain a globally convergent algorithm for solving large-scale NLS problems.

The main restriction of this scheme is that the subspace onto which a given Newton step is projected must solve the Newton equations with a certain accuracy which is monitored by the termination condition. This is enough to guarantee convergence.

This theory is sufficiently general that is hold for any algorithm that projects the problem on a lower dimensional subspace. The convergence results indicate promise for this research to solve large-scale NLS problems. Our next step is to investigate the performance of this algorithm on some NLS problems. The results will be reported in our forthcoming paper.

The authors would like to thank the editor-in-chief and the anonymous referees for their comments and useful suggestions in the improvement of the final form of this paper.