Computing the Pseudoinverse of Specific Toeplitz Matrices Using Rank-One Updates

Application of the pure rank-one update algorithm as well as a combination of rank-one updates and the Sherman-Morrison formula in computing theMoore-Penrose inverse of the particular Toeplitzmatrix is investigated in the present paper. Such Toeplitz matrices appear in the image restoration process and in many scientific areas that use the convolution. Four different approaches are developed, implemented, and tested on a number of numerical experiments.


Introduction
Let C × and C ×  denote the set of all complex  ×  matrices and the set of all complex  ×  matrices of rank , respectively.The identity matrix of an appropriate order is denoted by .The conjugate transpose, the range, the rank, and the null space of  ∈ C × are denoted by  * , R(), rank(), and N(), respectively.
Representation and computation of various generalized inverses are closely related to the following Penrose equations:  The set of all matrices obeying the conditions contained in a subset S ⊆ {1, 2, 3, 4} is denoted by {S}.Any matrix from {S} is called S-inverse of  and is denoted by  (S) .
By {S}  we denote the set of all S-inverses of  with rank .For any matrix  there exists a single element in the set {1, 2, 3, 4}, called the Moore-Penrose inverse of  and denoted by  † .
A rank-one modification of a matrix  ∈ C × is the matrix  = + * , which is created from  and two vectors  ∈ C  and  ∈ C  .The Sherman-Morrison formula (S-M shortly) gives the basic relationship between the inverses  −1 and  −1 (for more details see, e.g., [1]): The identity (2) provides a numerically efficient way to compute the inverse  −1 of the rank-one update .The S-M formula is important in many different fields of numerical computation; see for example [2][3][4][5][6][7].
On the other hand, Toeplitz matrices arise in a number of various theoretical investigations and applications.A number of iterative processes for finding generalized inverses of an arbitrary Toeplitz matrix by modifying Newton's method have been developed so far.The main results were stated in [8][9][10][11][12][13].Adaptations of the iterative processes to the Toeplitz structure are based on the usage of the displacement operator as well as the concept of displacement representation and displacement rank of matrices.
A variety of methods for computing the Moore-Penrose inverse of a rank-one modified matrix have been developed so far.Main results were derived in [14][15][16][17][18]. Relationships between various generalized inverses of an arbitrary matrix and corresponding generalized inverses of its rank-one modifications were investigated in [19].The leading idea in [15,16] was successive computation of the symmetric rank-one (SR1) updates ( −1 +    *  ) † of a given matrix , where  *  denotes the th row of  and  −1 = ∑ −1 =1    *  .The authors of the paper [15] introduced a computational procedure for the Moore-Penrose inverse of a symmetric rank-one perturbed matrix.Using this method, the authors of [16] proposed a finite method for computing the minimum-norm leastsquares solution of the linear system  = .
The results derived in [20] reveal that both the SR1 updates techniques and the S-M recursive rule are useful tools in the computation of various matrix products involving the Moore-Penrose inverse of certain symmetric matrices.Particularly, the algorithms introduced in [20] are numerically efficient in computation of {2, 4} and {2, 3} inverses.
In the present paper, we investigate the possibilities to apply the SR1 update procedure and the S-M formula in the computation of the Moore-Penrose inverse of specific Toeplitz matrices that appear in the image restoration process.Our main motivation arises from the convenience to apply the SR1 update and the S-M procedure in removing the blur which is always present in digital images.Firstly, both the SR1 update and the S-M formula are based on the usage of columns (or rows) of the input matrix.On the other hand, the matrices which appear in the mathematical model of blur in computer-generated images possess very specific structure which can be used to accelerate SR1 and S-M procedures.Namely, entries in Toeplitz matrices are constant along main diagonal parallels and, moreover, possess a significant proportion of zero elements.
The paper is organized as follows.Some basic notations and necessary facts are restated in Section 2. Also, some additional motivation is presented in the same section.Usage of the pure SR1 update algorithm, proposed in [15], in the computation of the Moore-Penrose inverse of a kind of Toeplitz matrices is considered in Section 3. A hybrid combination of the SR1 and the S-M recursive rules is defined in Section 4.An improvement of the SR1 procedure, which is derived on the basis of the specific structure of the underlying Toeplitz matrix, is presented in the same section.An application of introduced methods in image restoration is presented in Section 5.

Preliminaries and Motivation
Toeplitz matrices or diagonally constant matrices are matrices having constant diagonal entries.Toeplitz matrices which are applicable in the image restoration process contain ℓ nonzero main diagonal parallels above the main diagonal, where ℓ defines the blurring process.In what follows, let us consider the Toeplitz matrix of such form: The assumption  1 ̸ = 0 is active.To clarify notation, Toeplitz matrices of the general form (3) will be denoted shortly by We investigate the use of the SR1 update method, as described in [15, Algorithm 2], during the numerical computation of the Moore-Penrose inverse of Toeplitz matrices satisfying the Tpl ({ 1 }, { 1 , . . .,  ℓ }) pattern.Also, we examine different improvements of the original method.The improvements are based on appropriate adaptations of the SR1 method and the S-M formula to the characteristic structure of underlying matrices of type Tpl ({ 1 }, { 1 , . . .,  ℓ }).The method of SR1 updates is based on the expression which computes the Moore-Penrose inverse of the first  columns of the initial matrix  using the Moore-Penrose inverse of its first  − 1 columns.In detail, the SR1 method from [15] starts from the well-known representation  † = ( * ) †  * of the Moore-Penrose.If the th row of  is denoted by  *  , then Chen and Ji in [15] defined the matrix sequences   and   , as Clearly,   =  −1 +    *  is the rank-one modification of  −1 and Recall that the Moore-Penrose inverse of a general rankone modified matrix  =  +  * , where  is an arbitrary  [14] suggests six different cases that one has to follow in order to establish a relation between  † and  † .In [15], the authors proved that the real number   = 1 +  *   † −1   , corresponding to the term 1 +  *  −1  in (2), satisfies   ≥ 1.Later using this result in conjunction with the fact that   is a positive semidefinite matrix, the six cases of Theorem 3.1.3from [14] can be reduced to the two-case problem.This reduction simplifies the SR1 updates formulas.
Let us denote the first  columns of  by Also, the last  −  columns of  are denoted by {+1}  {} .Then the matrix  is given in the block form where the square block is the nonsingular band Toeplitz matrix and collects the last  −  columns of .
An application of Greville's partitioning method from [21] and the block partitioning method (BP method, shortly) from [22] in computing the Moore-Penrose inverse of the matrix  is presented in [23].According to Algorithms 1 and 2 and Lemma 3 from [23], it is clear that the specific structure of matrix  enables computation  † simply by computing the inverse of the nonsingular block  {} .
In this paper, we investigate some alternative methods for computing  † using the SR1 updates and the S-M formula.Our intention is to decrease computational complexity as much as possible using a specific structure of Toeplitz matrices of the general form Tpl ({ 1 }, { 1 , . . .,  ℓ }).

Computing the Pseudoinverse of a Toeplitz Matrix by Rank-One Updates
Our first attempt consists in applying the unmodified SR1 method from [15] in order to compute the pseudoinverse of the matrix  that belongs to the class Tpl ({ 1 }, { 1 , . . .,  ℓ }).
Obtained results are compared with corresponding results derived by applying Algorithm 2 from [23] (the BP method).The results of this comparison are presented in Table 1, where , ℓ, and  are the parameters which define the Gaussian blur modeled by the Toeplitz matrix .In the rest of the paper, it is assumed that the matrix  is of the order  × , where  =  + ℓ − 1 and ℓ ≥ 1 represents the width of the blurring function.
For the sake of simplicity, let us denote Algorithm 2 from [15] by SR1 and Algorithm 2 from [23] by BP.
From Table 1, it is observable that the SR1 method is marginally efficient in terms of accuracy, but it is time consuming, especially for larger dimensions.There is a theoretical explanation for these results.The number of multiplications and divisions included in the SR1 algorithm is about ( 2  + ) in the case  <  and ( 2  + ) in the case  < .Therefore, as it is stated in [15], the rank-one updates algorithm works efficiently in cases where it holds  ≪  or  ≫ .In our case,  =  + ℓ − 1, where ℓ ≥ 1 represents the width of the blurring process.Since, in the general case, ℓ is a relatively small integer, the conditions  <  and  ≈  are satisfied.This fact causes relative inefficiency of the SR1 method.
Our second attempt is to apply Algorithm SR1 to the matrix  {} , from (10), in order to generate its inverse.Then the Moore-Penrose inverse of  comes from the block partitioning method (called BP), as it was described in [23].This approach is shortly denoted by HSR1 + BP.
It is observable from Table 2 that the method HSR1 + BP is marginally efficient in terms of accuracy, but it is time consuming; see, for example, the case  = 400.
From the previous analysis of data included in Tables 1 and 2, we conclude that both algorithms SR1 and HSR1 + BP are not efficient approaches for computing  † with respect to the BP method in terms of computational speed, especially for large matrices.But both approaches show marginal efficiency in terms of the accuracy.

A Combination of SR1 Updates and S-M Formula
Two additional algorithms are defined in the current section.
The first one uses the SR1 updates to compute the matrix   =  {} ( {} ) * in the first step and the S-M formula to compute  †  in the subsequent step.The second algorithm replaces the usage of the SR1 updates from the first step by a direct construction of the matrix   .The matrix   can be generated efficiently in view of the specific structure of the input matrix .

SR1 Updates in Conjunction with the S-M Formula.
As usual, the th column of  is denoted by ℎ  .Then, since  is of full row rank, it is advantageous to define a proper SR1 recursive computational method for generating the Moore-Penrose inverse of  on the basis of the relation For that purpose, it is necessary to define the matrix sequence Lemma 1 reveals some of basic properties of the sequence   .Following the notation from [20], the notation ℎ  ∈ L(ℎ 1 , . . ., ℎ −1 ) means that a column ℎ  is linearly dependent on the previous columns ℎ 1 , . . ., ℎ −1 and ℎ  ∉ L(ℎ 1 , . . ., ℎ −1 ) indicates that ℎ  is linearly independent of ℎ 1 , . . ., ℎ −1 .
(iv) The following concluding relation is valid: Proof.(i) This part of the proof is implied by the fact that the columns of  are linearly independent.
(ii) According to the part (i), immediately follows rank( {} ) = rank( {−1} ) + 1, for each 1 <  ≤ .Later, it is easy to verify which implies rank (iii) This part of the proof can be verified by using (i) and (ii), together with the fact that   are all  ×  matrices, for each  = 1, . . ., .
(iv) According to part (iii), matrices   ,  ≤  ≤ , are invertible.Since is invertible, then it holds which completes the proof.
Theorem 5. Let the matrix  be defined as in (3), the matrix sequence   defined in (14), and the matrix sequence   defined in (20).Then the following recursive relations are true: Mathematical Problems in Engineering Also, matrices   ,  =  + 1, . . .,  are defined as So, ( 23) holds.
According to Theorem 5, we present Algorithm 1 for computing the Moore-Penrose inverse of the Toeplitz matrix .The algorithm is based on the SR1 modifications to compute   and subsequently the S-M formula to compute  † .For this purpose, we use the abbreviation HSR1 + S-M to denote Algorithm 1.
A Matlab implementation of Algorithm 1 is placed in Appendix.

An Improvement of the Hybrid Combination in Algorithm 1.
By taking into account the fact that the Toeplitz matrix  has a specific structure, the matrix   can be generated in an efficient way.In this way, an improvement of step (1) of Algorithm 1 is introduced.The effectiveness of this method will be justified in the numerical experiments presented in Section 4.3.
According to the proper structure of the matrix  and step (1) of Algorithm 1, the general structure of the matrix   is defined as Let  = ( 1 , . . .,   ) be one-dimensional vector; then the symmetric Toeplitz matrix generated by  is denoted by Toeplitz(): Also, for two appropriate vectors , V, we denote by  = Hankel(, V) the (symmetric) Hankel matrix whose first column is  and whose last row is V.
Theorem 6.The matrix   can be expressed as the difference of a specific banded symmetric Toeplitz matrix and a partitioned matrix of the form where the block  ℓ−1 is defined by  ℓ−1 =  2 , for an appropriate Hankel matrix .Proof.Let  and V be vectors defined by respectively.Then we define the following matrices: A simple verification shows that   =  − .
Next we present an example, in low dimensions, in order to illustrate Theorem 6.
According to Theorem 6 we present the following improved version of Algorithm 1, called Algorithm 2. A Matlab implementation of Algorithm 2 is placed in Appendix.It is appropriate to use the term IHSR1 to denote the improvement of the HSR1 step of Algorithm 1 and IHSR1 + S-M to denote Algorithm 2.
Note that proposed Algorithms 1 and 2 are not using directly, at any step, the Sherman-Morrison formula to compute the inverse of some matrix.In Section 4.4, we provide more information regarding this situation.

Numerical Experiments.
In this subsection we analyze numerical data arising during the computation of the Moore-Penrose inverse of the Toeplitz matrix  by applying a Matlab implementation of Algorithms 1 and 2. In order to test the time efficiency as well as the accuracy of considered methods, we enrich our collection of matrices used in Tables  1 and 2 with larger matrices.In order to complete our numerical study, we also compare the results derived by applying Algorithm 2 from [23] (Algorithm BP).A comparison of Algorithms 1 and 2 and BP is presented in Table 3.
From Table 3, it is observable that the level of improvement in CPU time that can be achieved by using Algorithm 2 instead of Algorithm 1 is significant.Also, it is clear that the accuracy of both algorithms is almost the same.Clearly, the BP method requires minimal CPU time.

Rounding Error Analysis.
The starting motivation of this subsection is the following basic problem: approximation error grows with repeated use of the S-M formula.As a consequence, computations which involve the S-M rule become unstable.For this purpose, it is interesting to investigate the error curves corresponding to the four Penrose equations during the iterations included in step (3) of Algorithm 2. Nevertheless, it is important to note that the proposed Algorithm 2 does not use in each step the S-M rule.Of course, Theorem 5 is in the heart of Algorithm 2 but also it is clear that the poor stability of the S-M formula has no serious influence on the stability of proposed algorithms.
So, in this subsection, we shall present error estimations regarding Algorithm 2. That is, we record and investigate the residual matrix norms  1 ,  2 ,  3 , and  4 , corresponding to the four Penrose equations during the execution of step (3) of Algorithm 2. The results are presented in Figures 1, 2 and the matrix   is symmetric, the matrix    is symmetric.For this reason, the graph corresponding to these values is almost a constant line.
The stable convergence is observable from each of these figures.Also, it is possible to notice very fast convergence in the terminal phase of the convergence as well as a relatively slower convergence in the middle of the recurrent process.This fact is understandable if one takes into account that it is necessary to take into consideration all the columns of the input matrix in order to get the complete information on the pseudoinverse.

Application in Image Restoration
Several image restoration examples are presented in this subsection.Experiments are done using Matlab programming package on an Intel(R) Core(TM) i5-2540M CPU @ 2.6 GHz 64-bit system with 8 GB RAM memory.
Numerical results corresponding to the Moore-Penrose inverse are derived using Algorithm 2. Results are derived using  = 1200 and  = 100 and different values of ℓ.Suppose that the matrix  ∈ R × corresponds to the original image and  ∈ R × is the matrix corresponding to the degraded image.
The model of the nonuniform blurring is the same as in [23] and assumes that the blurring of columns in the image is independent with respect to the blurring of its rows.Therefore, the relations between the original and the blurred image are expressed by the matrix equation where  =  + ℓ  − 1,  =  + ℓ  − 1, ℓ  is length of the vertical blurring, and ℓ  is length of the horizontal blurring (in pixels).Following the approach used in [23], the Moore-Penrose inverse approach is used to restore the blurred image , which produces the next approximation of : The algorithm for computing the Moore-Penrose inverse of the blurring matrix used in [24,25] is based on the method for computing the Moore-Penrose inverse of a full rank matrix, introduced in [26,27].In our case, we use Algorithm 2. The first approach is based on the usage of the pure SR1 update algorithm, proposed in [15].The second approach starts with the SR1 update algorithm and, after its completion, continues with the BP method.The third approach is a hybrid combination which comprises the SR1 updates in the first * = , () * = .

Table 1 :
Comparison of the rank-one updates method and the BP method.

Table 2 :
Comparison of the combination of AlgSR1 and AlgBP methods and the block partitioning method.
If an approximation of  † is denoted by , then the residual norms are denoted by