A Representation of Nonhomogeneous Quadratic Forms with Application to the Least Squares Solution

The least squares problem appears, among others, in linear models, and it refers to inconsistent system of linear equations. A crucial question is how to reduce the least squares solution in such a system to the usual solution in a consistent one. Traditionally, this is reached by differential calculus. We present a purely algebraic approach to this problem based on some identities for nonhomogeneous quadratic forms.


Introduction and Notation
The least squares problem appears, among others, in linear models, and it refers to inconsistent system Ax b of linear equations.Formally, it reduces to minimizing the nonhomogeneous quadratic form Classical approach to the problem, presented in such known books as Scheffé 1, Chapter 1 , Rao 2, pages 222 and 223 , Rao and Toutenburg 3, pages 20-23 , uses differential calculus and leads to the so called normal equation A T Ax A T b, which is consistent.The aim of this note is to present some useful algebraic identities for nonhomogeneous quadratic forms leading directly to normal equation.Traditional vector-matrix notation will be used.Among others, if M is a matrix then M T , R M , and r M stand for its transposition, range column space , and rank.Moreover, by R n will be denoted the n-dimensional euclidean space represented by column vectors.

Background
Any system of linear equations may be presented in the vector-matrix form as Ax b, 2.
where A is a given n × p matrix, b is a given vector in R n , while x ∈ R p is unknown vector.It is well known that 2.1 is consistent, if and only if, b belongs to the range R A .If 2.1 is inconsistent, one can seek for a vector x minimizing the norm Ax − b or, equivalently, its square Ax − b T Ax − b .The Least Squares Solution (LSS) of 2.1 is defined as a vector x 0 ∈ R p such that A crucial problem is how to reduce the LSS of the inconsistent equation 2.1 to the usual solution of a consistent one.Formally, the least squares problem deals with minimizing the nonhomogeneous quadratic form f x Ax − b T Ax − b .Traditionally, this problem is solved by differential calculus and leads to the normal equation A T Ax A T b.
In the next section, we will present some useful algebraic identities for nonhomogeneous quadratic forms.They yield directly the inequality 2.2 .

Identities and Inequalities for Nonhomogeneous Quadratic Forms
The usual, that is homogeneous quadratic form is a real function f M x x T Mx defined on R p .In this note, we shall consider also nonhomogeneous quadratic forms of type where M is a symmetric p × p matrix and a is a vector in R p .Some inequalities for nonhomogeneous quadratic forms may be found in St epniak 4 .Let us recall one of these results, which is very useful in the nonhomogeneous linear estimation.Lemma 3.1.For any symmetric nonnegative definite matrices M 1 and M 2 of order p, the condition for some c 1 , c 2 ∈ R, x 1 ,x 2 ∈ R p and all x ∈ R p implies that M 1 − M 2 is nonnegative definite and c 1 − c 2 ≥ 0. Now we will present some identity which may serve as a convenient tool in the LSS of 2.1 .For convenience, we will start from the case r A p, leaving the singular case r A < p to Section 5.