Does the Best-Fitting Curve Always Exist ?

In many areas of human practice, one needs to approximate a set of points P1, . . . , Pn ∈ R representing experimental data or observations by a simple geometric shape, such as a line, plane, circular arc, elliptic arc, and spherical surface. This problem is known as fitting amodel object line, circle, sphere, etc. to observed data points. The best fit is achieved when the geometric distances from the given points to the model object are minimized, in the least squares sense. Finding the best fit reduces to the minimization of the objective function


Introduction
In many areas of human practice, one needs to approximate a set of points P 1 , . . ., P n ∈ R d representing experimental data or observations by a simple geometric shape, such as a line, plane, circular arc, elliptic arc, and spherical surface.This problem is known as fitting a model object line, circle, sphere, etc. to observed data points.
The best fit is achieved when the geometric distances from the given points to the model object are minimized, in the least squares sense.Finding the best fit reduces to the minimization of the objective function where S ⊂ R d denotes the model object line, circle, sphere, etc. .While other fitting criteria are used as well, it is the minimum of 1.1 that is universally recognized as the most desirable solution of the fitting problem.It has been adopted as a standard in industrial applications 1, 2 .In statistics, the minimum of 1.1 corresponds to the maximum likelihood estimate under usual assumptions known as functional model; see 3, 4 .Minimization of 1.1 is also called orthogonal distance regression ODR .
Geometric fitting in the sense of minimizing 1.1 has a long history.First publications on fitting linear objects lines, planes, etc. to given points in 2D and 3D date back to the late nineteenth century 5 .The problem of fitting lines and planes was solved analytically in the 1870s 6 , and the statistical properties of the resulting estimates were studies throughout the twenties century 7-9 , the most important discoveries being made in the 1970s 3, 10, 11 .These studies gave rise to a new branch of mathematical statistics now known as errors-invariable EIV regression analysis 3, 4 .
Since the 1950s, fitting circles and spheres to data points became popular in archaeology, industry, and so forth, 12, 13 .In the 1970s, researchers started fitting ellipses and hyperbolas to data points 14-17 .The interest to the problem of geometric fitting surged in the 1990s when it became an agenda issue for the rapidly growing computer science community.Indeed, fitting simple contours to digitized images is one of the basic tasks in pattern recognition and computer vision.In most cases, those contours are lines, circles, ellipses and other conic sections called simply conics, for brevity in 2D, and planes, spheres, ellipsoids in 3D.More complicated curves and surfaces are used occasionally too 1, 18 , but it is more common to divide complex images into smaller segments and approximate each of those by a line or by a circular arc.This way one can approximate a complex contour by a polygonal line or a circular spline see, e.g., 19, 20 .Most publications on the geometric fitting problem are devoted to analytic solutions which are only possible for lines and planes , or practical algorithms for finding the bestfitting object, or statistical properties of the resulting estimates.Very rarely one addresses fundamental issues such as the existence and uniqueness of the best fit.If these issues do come up, one either assumes that the best fit exists and is unique, or just points out examples to the contrary without deep investigation.
In this paper we address the issue of existence of the best fit in a general and comprehensive manner.The issue of uniqueness will be treated in a separate paper.These issues turn out to be quite nontrivial and lead to unexpected conclusions.As a glimpse of our results, we provide Table 1 summarizing the state of affairs in the problem of fitting 2D objects here Yes means the best-fitting object exists or is unique in all respective cases; No means the existence/uniqueness fails in some of the respective cases .
We see that the existence and uniqueness of the best-fitting object cannot be just taken for granted.Actually 2/3 of the answers in Table 1 are negative.The uniqueness can never be guaranteed.The existence is guaranteed only for lines.In typical cases i.e., for typical sets of data points, in probabilistic sense; see precise definition in Section 6 , the best-fitting line and circle do exist, but the best-fitting ellipse does not.
The existence and uniqueness of the best fit are not only of theoretical interest but also practically relevant.For example, knowing under what conditions the problem does not have a solution might help us understand why the computer algorithm keeps diverging, or returns nonsense, or crashes altogether.While the cases where the best fit does not theoretically exist may be exceptional, nearby cases may be practically hard to handle, as the best-fitting object may be extremely difficult to find.
The nonuniqueness has its practical implications too.It means that the best-fitting object may not be stable under slight perturbations of the data points.An example is described by Nievergelt 21 ; he presented a set of n 4 points that can be fitted by three different circles equally well.Then by arbitrarily small changes in the coordinates of the points one can make any of these three circles fit the points a bit better than the other two circles, thus the best-fitting circle will change abruptly.See also a similar example in 4, Section 2.2 .
Here we develop a general approach to the studies of existence of the best fit.Our motivation primarily comes from popular applications where one fits lines, circles, ellipses, and other conics, but our methods and ideas can be applied to much more general models.Our approach works in any dimension; that is, we can treat data points P 1 , . . ., P n ∈ R d and model objects S ⊂ R d for any d ≥ 2, but for the sake of notational simplicity and ease of illustrations we mostly restrict the exposition to the planar case d 2.

W-Convergence and the Induced Topology
A crucial notion in our analysis is that of convergence of model objects .

Motivating Example
Consider a sequence of lines L n {y x/n}.They all pass through the origin 0, 0 and their slopes 1/n decrease as n grows Figure 1 .Naturally, we would consider this sequence as convergent; it converges to the x-axis, that is, to the line L {y 0}, as n → ∞.
However, it is not easy to define convergence so that the above sequence of lines would satisfy it.For example, we may try to use Hausdorff distance to measure how close two objects S 1 and S 2 ⊂ R 2 are.It is defined by dist H S 1 , S 2 max sup

2.1
If the two sets S 1 and S 2 are closed and the Hausdorff distance between them is zero, that is, dist H S 1 , S 2 0, then they coincide S 1 S 2 .If the Hausdorff distance is small, the two sets S 1 and S 2 nearly coincide with each other.So the Hausdorff distance seems appropriate for our purposes.But it turns out that the Hausdorff distance between the lines L n and L in our example is infinite, that is, dist H L n , L ∞ for every n.Thus the Hausdorff distance cannot be used to characterize convergence of lines as we would like to see it.

Windows
So why do we think that the line L n converges to the line L, despite an infinite Hausdorff distance between them?It is because we do not really see an infinite line, we only "see" objects in a certain finite area, like in Figure 1.Suppose we see objects in some rectangle which for the moment will play the role of our "window" through which we look at the plane.
Then we see segments of our lines within R; that is, we see intersections L n ∩ R and L ∩ R. Now clearly the segment L n ∩ R gets closer to L ∩ R, as n grows, and in the limit n → ∞ they become identical.This is why we see the lines L n converging to L. We see this convergence no matter how large the window R is.Note that the Hausdorff distance between L n ∩ R and

Window-Restricted Hausdorff Distance
Taking cue from the above example, we define the Hausdorff distance between sets S 1 and S 2 within a finite window R as follows: dist H S 1 , S 2 ; R max sup

2.3
The formula 2.3 applies whenever both sets, S 1 and S 2 , intersect the window R. If only one set, say S 1 , intersects R, we modify 2.3 as follows: dist H S 1 , S 2 ; R sup A similar modification is used if only S 2 intersects R. If neither set intersects the window R, we simply set dist H S 1 , S 2 ; R 0 because we "see" two empty sets, which are not distinguishable .
We now define our notion of convergence for a sequence of sets.
Definition 2.1.Let S n ⊂ R 2 be some sets and S ⊂ R 2 another set.We say that the sequence S n converges to S if for any finite window R we have dist H S n , S; R −→ 0 as n −→ ∞.

2.5
In this definition the use of finite windows is crucial; hence we will call the resulting notion Window convergence, or W convergence, for short.According to this definition, the sequence of lines L n in the above example indeed converges to the limit line L.
For finite-size bounded objects, like circles or ellipses, the W-convergence is equivalent to the convergence with respect to the Hausdorff distance.However for unbounded objects, such as lines and hyperbolas, we have to use our window-restricted Hausdorff distance and formula 2.5 .
Our definition of convergence i.e., W-convergence is intuitively clear, but some complications may arise if it is used too widely.For example, if the limit object S is not necessarily closed, then the same sequence {S n } may have more than one limit S though the closure S of every limit set S will be the same .Thus, from now on, to avoid pathological situations, we will assume that all our sets S ⊂ R 2 are closed.All the standard model objectslines, circles, ellipses, hyperbolas-are closed.
Convergence induces topology in the collection of all closed sets S ⊂ R 2 , which we denote by X.A set Y ⊂ X is closed if for any sequence of sets S n ∈ Y converging to a limit set S the limit set also belongs to The above topology on X is metrizable.This means we can define a metric on X, that is, a distance between closed sets in R 2 , in such a way that the W-convergence S n → S means exactly that the distance between S n and S goes down to zero.Such a distance can be defined in many ways; here is one of them.
Definition 2.2.The W distance or Window distance between two closed sets S 1 and S 2 ⊂ R 2 is defined as follows: where R k is a square window of size 2k × 2k, that is, R k {|x| ≤ k, |y| ≤ k}.
In this formula, we use a growing sequence of nested windows and the Hausdorff distances between S 1 and S 2 within those windows balanced by the factors 2 −k .In the formula 2.6 , the first nonzero term corresponds to the smallest window R k that intersects at least one of the two sets, S 1 or S 2 .The sum in 2.6 is always finite.Indeed, let us suppose, for simplicity, that both S 1 and S 2 intersect each window R k .Then since the distance between any two points in R k is at most 2 √ 2k the above sum is bounded by 2

Continuity of Objective Function
Recall finding the best-fitting object S for a given set of points P 1 , . . ., P n ∈ R 2 consists of minimization of the objective function 1.1 .Let M denote the collection of model objects.We put no restrictions on M other than all S ∈ M are assumed to be closed sets.The collection M is then a subset of the space X of all closed subsets S ⊂ R 2 .The topology and metric on X defined above now induce a topology and metric on M.

Redundancy Principle
If an object S ∈ M is a subset of another object S ∈ M, that is, S ⊂ S, then for any point P we have dist P, S ≤ dist P, S ; thus S cannot fit any set of data points better than S does.So for the purpose of minimizing F, that is, finding the best-fitting object, we may ignore all model objects that are proper subsets of other model objects they are redundant .This may reduce the collection M somewhat.Such a reduction is not necessary, it is just a matter of convenience, and we will occasionally apply it below.
Conversely, there is no harm in considering any subset S ⊂ S of an object S ∈ M as a smaller object, too.Indeed, if S provides a best fit i.e., minimizes the objective function F , then so does S, because F S ≤ F S .Hence including S into the collection M will not really be an extension of M, its inclusion will not change the best fit.Theorem 3.1 continuity of F .For any given points P 1 , . . ., P n and any collection M of model objects the function F defined by 1.1 that is continuous on M.This means that if a sequence of objects S m ∈ M converges (i.e., W converges) to another object S ∈ M, then F S m → F S .
Proof.Since F S is the sum of squares of distances dist P i , S to individual points P i , see 1.1 , it is enough to verify that the distance dist P, S is a continuous function of S for any given point P .
Suppose P ∈ R 2 and a sequence of closed sets S m W converging to a closed set S. We denote by Q ∈ S the point in S closest to P , that is, such that dist P, Q dist P, S ; such a point Q exists because S is closed.Denote by D the disk centered on P of radius 1 dist P, Q ; it contains Q.Let R be a window containing the disk D.
Since R contains Q, it intersects with S, that is, R ∩ S / ∅.This guarantees that dist H S m , S; R → 0. Thus, there are points Q m ∈ S m such that Q m → Q.Since dist P, S m ≤ dist P, Q m , we conclude that the upper limit of the sequence dist P, S m does not exceed dist P, S , that is, lim sup dist P, S m ≤ dist P, S .

3.1
On the other hand, we will show that the lower limit of the sequence dist P, S m cannot be smaller than dist P, S , that is, lim inf dist P, S m ≥ dist P, S .

3.2
The estimates 3.1 and 3.2 together imply that dist P, S m → dist P, S , as desired, and hence the distance function dist P, S will be continuous on M. It remains to prove 3.2 .
To prove 3.2 we assume that it is false.Then there is a subsequence S m k in our sequence of sets S m such that lim Denote by

3.6
The limit of the first term on the right hand side of 3.6 is < dist P, S by 3.4 , and the limit of the second term is zero by 3.5 .This implies dist P, S < dist P, S , which is absurd.The contradiction proves 3.2 .And the proof of 3.2 completes the proof of the theorem.

Existence of the Best Fit
The best-fitting object S best ∈ M corresponds to the global minimum of the objective function F, that is, S best arg min S∈M F S .The function F defined by 1.1 cannot be negative; thus, it always has an infimum 4.1 This means that one cannot find objects S ∈ M such that F S < F 0 , but one can find objects S ∈ M such that F S is arbitrarily close to F 0 .More precisely, there is a sequence of objects S m such that F S m > F 0 for each m and F S m → F 0 as m → ∞.
In practical terms, one usually runs a computer algorithm that executes a certain iterative procedure.It produces a sequence of objects S m here m denotes the iteration number such that F S m < F S m−1 ; that is, the quality of approximations improves at every step.If the procedure is successful, the value F S m converges to the minimal possible value, F 0 , and the sequence of objects S m converges i.e., W-converges to some limit object S 0 .Then the continuity of the objective function F proven earlier guarantees that F S 0 F 0 ; that is, S 0 indeed provides the global minimum of the objective function, so it is the best-fitting object S 0 S best .
A problem arises if the limit object S 0 does not belong to the given collection M; hence it is not admissible.Then we end up with a sequence of objects S m , each of which fits approximates the given points better than the previous one, but not as good as the next one.None of them would qualify as the best fit, and in fact the best fit would not exist, so the fitting problem would have no solution.

Illustrative Example
Suppose that our model objects are circles, and our given points are P 1 −1, 0 , P 2 0, 0 and P 3 1, 0 .Then the sequence of circles S m defined by x 2 y −m 2 m 2 will fit the points progressively better tighter as m grows, so that F S m → 0 as m → ∞ Figure 2 .On the other hand, no circle can pass through three collinear points; hence no circle S satisfies F S 0. Thus the circle-fitting problem has no solution.
The above sequence of circles S m converges in our terms, W-converges to the x axis, which is a line, so it is natural to declare that line "the best fit."This may not be satisfactory in some practical applications, where one really needs to produce an estimate of the circle's center and radius.But if we want to present the best-fitting object here, it is clearly and undeniably the line y 0.
Thus, in order to guarantee the existence of the best-fitting object in all cases, we need to include in our collection M all objects that can be obtained as limits of sequences of objects from M. Such "limit objects" are called limit points of M, in the language of topology.Definition 4.1.A collection M which already contains all its "limit points" is said to be closed.If the collection M is not closed, then an extended collection M that includes all the limit points of M is called the closure of M.
For example, the collection M L of all lines in R 2 is closed, as a sequence of lines can only converge to a line.The collection M C of all circles in R 2 is not closed, as a sequence of circles may converge to a line.The closure of the collection of circles M C includes all circles and all lines, that is, Strictly speaking, a sequence of circles may also converge to a single point, see Section 5, so singletons need to be included too.We see that the collection M of model objects must be closed if we want the best-fitting object to exist in all cases.If M is not closed, we have to extend it by adding all its limit points and thus make it closed.Theorem 4.2 existence of the best fit .Suppose that the given collection M of model objects is closed.Then for any given points P 1 , . . ., P n there exists the best-fitting object S best ∈ M, that is, the objective function F attains its global minimum on M.
Proof.The key ingredients of our proof will be the continuity of the objective function F and the compactness of a restricted domain of that function.We recall that a continuous real-valued function on a compact set always takes its maximum value and its minimum value on that set.In metric spaces like our M , a subset M 0 ⊂ M is compact if every sequence of its elements S m ∈ M 0 has a subsequence S m k that converges to another element S ∈ M 0 , that is, S m k → S as k → ∞.
We recall that our objective function F is continuous, and its domain M is now assumed to be closed.If it was compact, the above general fact would guarantee that F had a global minimum, as desired.But in most practical settings M is not compact.Indeed, if it was compact, the function F would not only have a minimum, but also a maximum.And this is usually impossible; one can usually find model objects arbitrarily far from the given points, thus making the distances from the points to the object arbitrarily large.Thus we need to find a smaller restricted subcollection M 0 ⊂ M which will be compact, and then we will apply the above general fact.
Let D r {x 2 y 2 ≤ r 2 } denote the disk of radius r centered on the origin 0, 0 .We can find a large enough disk D r that satisfies two conditions: i D r covers all the given points P 1 , . . ., P n and ii D r intersects at least one object S 0 ∈ M, that is, D∩S 0 / ∅.The distances from the given points to S 0 cannot exceed the diameter of D r , which is 2r, hence F S 0 ≤ 2r 2 n.Now we define our subcollection M 0 ⊂ M; it consists of all model objects S ∈ M that intersect the larger disk D 3r of radius 3r.Objects that lie entirely outside D 3r are not included in M 0 .Note that the subcollection M 0 contains at least one object S 0 mentioned above ; hence M 0 is not empty.
Recall that all our given points P 1 , . . ., P n lie in D r .They are separated from the region outside the larger disk D 3r by a "no man's land"-the ring D 3r \ D r , which is 2r wide.Thus the distances from the given points to any object S which was not included in M 0 are greater than 2r; hence for such objects we have F S > 2r 2 n.So objects not included in M 0 cannot fit our points better than S 0 does.Thus they can be ignored in the process of minimization of F.More precisely, if we find the best-fitting object S best within the subcollection M 0 , then for any other object S ∈ M \ M 0 we will have which shows that S best will be also the best-fitting object within the entire collection M.
It remains to verify that the subcollection M 0 is compact; that is, every sequence of objects S m ∈ M 0 has a subsequence converging to another object S * ∈ M 0 .We will use the following standard fact.In a compact metric space any sequence of compact subsets has a convergent subsequence with respect to the Hausdorff metric.Now let j 3r 1 be the smallest integer greater than 3r.Recall that all the objects in M 0 are required to intersect D 3r ; thus, they all intersect D j as well.The sets S m ∩ D j are compact.By the above general fact, there is a subsequence S that converges in the larger disk ∩ D j , S * j → 0, we see that S * j 1 ∩ D j S * j ; that is, the limit sets S * j and S * j 1 "agree" within D j .Then we continue this procedure inductively for the progressively larger disks D j 2 , D j 3 , . ... In the end we use standard Cantor's diagonal argument to construct a single subsequence S m k such that for every i ≥ j we have dist H S m k ∩ D i , S * i → 0 as k → ∞, and the limiting subsets S * i ⊂ D i "agree" in the sense S * i 1 ∩ D i S * i for every i ≥ j.Then it follows that the sequence of objects S m k converges i.e., W converges to the closed set S * ∪ i≥j S * i .The limit set S * must belong to our collection M because that collection was assumed to be closed.Lastly, S * intersects the disk D 3r , so it also belongs to the subcollection M 0 .Our proof is now complete.

Lines and Planes
The most basic and oldest fitting application is fitting lines in 2D.The model collection M L consists of all lines L ⊂ R 2 .It is easy to see that a sequence of lines L m can only converge to another line; hence, the collection M L of lines is closed.This guarantees that the fitting problem always has a solution.
Similarly, the collection of lines in 3D is closed.The collection of planes in 3D is closed, too.Thus the problems of fitting lines and planes always have a solution.The same is true for the problem of fitting k-dimensional planes in a d-dimensional space R d for any 1 ≤ k < d.

Circles and Spheres
A less-trivial task is fitting circles in 2D.Now the model collection M C consists of all circles C ⊂ R 2 .A sequence of circles C m can converge to an object of three possible types: a circle C 0 ⊂ R 2 , a line L 0 ⊂ R 2 , or a single point singleton P 0 ∈ R 2 .Singletons can be regarded as degenerate circles whose radius is zero , or they can be ignored based on the redundancy principle introduced in Section 3.But the lines are objects of a different type; thus, the collection M C of circles is not closed.
As a result, the circle fitting problem does not always have a solution; a simple example was given in Section 4. In order to guarantee the existence of the best-fitting object, one needs to extend the given collection M C by adding all its "limit points," in this case-lines.The extended collection will include all circles and all lines, that is, 5.1 Now the extended circle fitting problem always has a solution; for any data points P 1 , . . ., P n there is a best-fitting object S best ∈ M C that minimizes the sum of squares of the distances to P 1 , . . ., P n .One has to keep in mind, though, that the best-fitting object may be a line, rather than a circle.The problem of fitting circles to data points has been around since the 1950's 13 .Occasional nonexistence of the best-fitting circle has been first noticed empirically; see, for example, Berman and Culpin 22 .The first theoretical analysis of the non-existence phenomenon was done by Nievergelt 21 who traced it to the noncompactness of the underlying model space and concluded that lines needed to be included in the model space to guarantee the existence of the best fit.He actually studied this phenomenon in many dimensions, so he dealt with hyperspheres and hyperplanes.He called hyperplanes "generalized hyperspheres."Similar analysis and conclusions, for circles and lines, were published later, independently, by Chernov and Lesort 23 and by Zelniker and Clarkson 24 ; see also a recent book 4 .

Ellipses
A popular task in computer vision is fitting ellipses.Now the model collection M E consists of all ellipses E ⊂ R 2 .A sequence of ellipses E m may converge to an object of many possible types: an ellipse, a parabola, a line, a pair of parallel lines, a segment of a line, a half-line a ray ; see Figure 3. Line segments can be regarded as degenerate ellipses whose minor axis is zero.Alternatively, line segments and half-lines can be ignored based on the redundancy principle Section 3 .Thus we have lines, pairs of parallel lines, and parabolas as limit objects of different types.
We see that the collection M E of ellipses is not closed.As a result, the ellipse-fitting problem does not always have a solution.For example, this happens when given points that are placed on a line or on a parabola.Actually, there are many more examples; see Section 6.
In order to guarantee the existence of the best-fitting object, one needs to extend the collection M E of ellipses by adding all its "limit points," in this case-lines, pairs of parallel lines, and parabolas.We denote that extended collection by M E that where M denotes the collection of pairs of parallel lines, and M ∪ the collection of parabolas the symbol ∪ resembles a parabola .Now the extended ellipse-fitting problem always has a solution, for any data points P 1 , . . ., P n there will be a best-fitting object S best ∈ M E that minimizes the sum of squares of the distances to P 1 , . . ., P n .One has to keep in mind, though, that the best-fitting object may be a line, or a pair of parallel lines, or a parabola, rather than an ellipse.
The ellipse fitting problem has been around since the 1970s.The need to deal with limiting cases was first notice by Bookstein 16 who wrote "The fitting of a parabola is a limiting case, exactly transitional between ellipse and hyperbola.As the center of ellipse moves off toward infinity while its major axis and the curvature of one end are held constant. ..." A first theoretical analysis on the non-existence of the best ellipse was done by Nievergelt in 25 who traced it to the noncompactness of the underlying model space and concluded that parabolas needed to be included in the model space to guarantee the existence of the best fit.

Conics
We now turn to a more general task of fitting quadratic curves also known as conic sections, or simply conics .Now the model collection M Q consists of all nondegenerate quadratic curves, by which we mean ellipses, parabolas, and hyperbolas.A sequence of conics may converge to an object of many possible types: a conic, a line, a line segment, a ray, two opposite rays, and a pair of parallel lines, and a pair of intersecting lines; see Figures 3 and 4. Every line segment, ray, and two opposite rays are a part subset of a full line, so these objects can be ignored based on the redundancy principle Section 3 .But pairs of parallel lines and pairs of intersecting lines constitute limit objects of new types.We see that the collection M Q of quadratic curves conics is not closed.As a result, the conic fitting problem does not always have a solution.
In order to guarantee the existence of the best-fitting object, one needs to extend the collection M Q of quadratic curves by adding all its "limit points"-pairs of lines which may be intersecting, parallel, or coincident where M LL denotes the collection of pairs of lines.Now the extended conic fitting problem always has a solution; for any data points P 1 , . . ., P n there is a best-fitting object S best ∈ M Q that minimizes the sum of squares of the distances to P 1 , . . ., P n .One has to keep in mind, though, that the best-fitting object may be a pair of lines, rather than a genuine conic.

Sufficient and Deficient Models
We see that if the collection M of adopted model objects is not closed, then in order to guarantee the existence of the best fit we have to close it up, that is, add to M all objects that are obtained as limits of existing objects.The new, extended collection M will be closed and will always provide the best fit.Here we discuss practical implications of extending M. Definition 6.1.The original objects S ∈ M are called primary objects and the objects that have been added, that is, S ∈ M \ M are called secondary objects.
The above extension of M by secondary objects, though necessary, may not be completely harmless.Suppose again that one needs to fit circles to observed points.Since the collection of circles M C is not closed, one has to add lines to it.That is, one really has to operate with the extended collection M C M C ∪ M L in which circles are primary objects and lines are secondary objects.The best-fitting object to a given set of points will then belong to M C ; hence occasionally it will be a line, rather than a circle.
In practical applications, though, a line may not be totally acceptable as the best-fitting object.For example, if one needs to produce an estimate of the circle's radius and center, a line will be of little help; it has no radius or center.Thus, when a secondary object happens to be the best fit to the given data points, various unwanted complications may arise.
Next we investigate how frequently such unwanted events occur.
In most applications points are obtained measured or observed with some random noise.The noise has a probability distribution, which in statistical studies is usually assumed to be normal 3, 4 .For our purposes it is enough to just assume that given points P 1 , . . ., P n have an absolutely continuous distribution: Assumption 6.2.The points P 1 , . . ., P n are obtained randomly with a probability distribution that has a joint probability density ρ x 1 , y 1 , . . ., x n , y n > 0.
Let us first see how frequently the best-fitting circle fails to exist under this assumption.We begin with the simplest case of n 3 points.If they are not collinear, then they can be interpolated by a circle.If they are collinear and distinct , there is no interpolating circle, so the best fit is achieved by line.The three points x i , y i , 1 ≤ i ≤ 3 are collinear if and only if This equation defines a hypersurface in R 6 .Under the above assumption, the probability of that hypersurface is zero; that is, the collinearity occurs with probability zero.In simple practical terms, it is "impossible," it "never happens."If we generate three data points by using a computer random number generator, we will practically never see collinear points.All our practical experience tells us; the best-fitting circle always exists.Perhaps for this reason the circle-fitting problem is usually studied without including lines in the collection of model objects.
In the case of n > 3 data points, the best-fitting circle fails to exist when the points are collinear, but there are other instances too.For example, let n 4 points be at 0, 1 , 0, −1 , A, 0 , and −B, 0 for some large A, B 1. Then it is not hard to check, by direct inspection, that the best-fitting circle fails to exist, and the best fit is achieved by the line y 0.
It is not easy to describe all sets of points for which the best-fitting circle fails to exist, but such a description exists-it is given in 4 see Theorem 8 on page 68 there .We skip technical details, only state the final conclusion; for every n > 3, the best-fitting circle exists unless the 2n -vector x 1 , y 1 , . . ., x n , y n belongs to a certain algebraic submanifold in R 2n .So, under our assumption, the best-fitting circle exists with probability one.
In other words, no matter how large or small the data set is, the best-fitting circle will exist with probability one, so practically we never have to resort to secondary objects lines ; that is, we never have to worry about existence.The model of circles is adequate and sufficient; it does not really require an extension.Definition 6.3.We say that a collection M of model objects is sufficient for fitting purposes if the best-fitting object S best exists with probability one under the above assumption.Otherwise the model collection will be called deficient.
We note that under our assumption an event occurs with probability zero if and only if it occurs on a subset A ⊂ R 2n of Lebesgue measure zero, that is, Leb A 0. Therefore, our notion of sufficiency does not depend on the choice of the probability density ρ.
As we have just seen, the collection of circles is sufficient for fitting purposes.Next we examine the model collection M E of ellipses E ⊂ R 2 .As we know, this collection is not closed; its closure must include all lines, pairs of parallel lines, and parabolas.In another experiment we sampled points from the unit square 0, 1 × 0, 1 with a uniform distribution; that is, we selected each coordinate x i and y i independently from the unit interval 0, 1 .In this experiment ellipses turned up with probability 28% and hyperbolas with probability 72%, again an overwhelming domination of hyperbolas!And again, these percentages will remain the same if we generate points x i , y i according to any other rectangular uniform distribution.Indeed, any rectangle can be transformed to the square 0, 1 × 0, 1 by a linear map of the plane, and then the same argument as before applies.
We also conducted numerical tests for n > 5 points.In these tests we generated points P 1 , . . ., P n , independently according to four predefined distributions: normal with covariance matrix V 1 1 0 0 1 , normal with covariance matrix V 2 4 0 0 1 , uniform in the square 0, 1 × 0, 1 , uniform in the rectangle 0, 2 × 0, 1 .Table 2 shows the percentage of randomly generated samples of n points for which the best-fitting ellipse exists.
We see that in most cases the best-fitting ellipse fails to exist with a high probability.Only for the uniform distribution is the square 0, 1 × 0, 1 the failures decrease as n → ∞, and apparently the probability of existence of the best-fitting ellipse grows to 100%.We will explain this in Section 7.
We note that for n ≥ 6 points there is no interpolating conic, so one has to find a global minimum of F via an extensive search over the model space M Q of conics.This is a very timeconsuming procedure and its results are never totally reliable.Thus our estimates, especially for larger n's, are quite approximate, but we believe that they are accurate to within 5%.

Insufficiency of Ellipses
We conclude that the collection of ellipses is not sufficient for fitting purposes.This means that there is a real chance that for a given set of data points no ellipse could be selected as the best fit to the points; that is, the ellipse-fitting problem would have no solution.More precisely, for any ellipse E there will be another ellipse E providing a better fit, in the sense F E < F E .If one constructs a sequence of ellipses that fit the given points progressively better and on which the objective function F converges to its infimum, then those ellipses will grow in size and converge to something different than an ellipse most likely, to a parabola .
In practical terms, one usually runs a computer algorithm that executes an iterative procedure such as Gauss-Newton or Levenberg-Marquardt.It produces a sequence of ellipses E m here m denotes the iteration number such that F E m < F E m−1 ; that is, the quality of approximations improves at every step, but those ellipses would keep growing in size and approach a parabola.This situation commonly occurs in practice 16, 25, 26 , and one either accepts a parabola as the best fit or choose an ellipse by an alternative procedure e.g., by sirect ellipse fit 27 .
We see that whenever the best-fitting ellipse fails to exist, the ellipse-fitting procedure will attempt to move beyond the collection of ellipses, so it will end up on the border of that collection and then return a secondary object a parabola or a pair of lines .In a sense, the scope of the collection of ellipses is too narrow for fitting purposes.In other words, this collection is deficient, or badly incomplete.Its deficiency indicates that it should be substantially extended for the fitting problem to have a reasonable adequate solution with probability one.

Conics
Such an extension is achieved by the collection of all conics ellipses, hyperbolas, and parabolas denoted by M Q ; see Section 5.If one searches for the best-fitting object S best in the entire collection of conics, then S best will exist with probability one.This was confirmed in numerous computer experiments that we have conducted.In fact, the best-fitting object has always been either an ellipse or hyperbola, so parabolas apparently can be excluded.Thus our numerical tests strongly suggest that the collection M EH M E ∪ M H of ellipses, and hyperbolas is sufficient though we do not have a proof .

Generalizations
In the basic formula 1.1 all the n points make equal contributions to F. In some applications, though, one needs to minimize a weighted sum for some fixed w i > 0. This problem is known as weighted least squares.All our results on the existence of the best fit apply to it as well.More generally, given a probability distribution dP x, y on R 2 , one can "fit" an object S to it or approximate it by minimizing where d x, y dist x, y , S .Note that 7.1 is a particular version of 7.2 in which dP is a discrete measure with a finite support.In order to guarantee the finiteness of the integral in 7.2 one needs to assume that the distribution dP has finite second moments.After that all our results apply.The generalization 7.2 is relevant to the numerical tests described in Section 6. Suppose again that n points are selected independently according to a probability distribution dP x, y on R 2 .As n grows, due to the law of large numbers of probability theory that the normalized objective function 1/n F converges exactly to the integral 7.2 .Now the global minimum of the function F P S would give us the best-fitting object corresponding to the given probability distribution dP, which would be the best "asymptotic" fit to a random sample of size n selected from the distribution dP, as n → ∞.This relation helps us explain some experimental results reported in Section 6. Suppose that dP is a uniform distribution in a rectangle R 0, L × 0, 1 .Then the integral in 7.2 becomes x, y dy dx.

7.3
We have computed it numerically in order to find the best-fitting conic S. The results are given below.
Perfect Square R 0, 1 × 0, 1 One would naturally expect that the best fit to a mass uniformly distributed in a perfect square would be either a circle or a pair of lines say, the diagonals of the square .What we found was totally unexpected; the best-fitting conic is an ellipse!Its center coincides with the center of the square, and its axes are aligned with the sides of the square, but its axes are different; the major axis is 2a 0.9489 and the minor axis is 2b 0.6445 assuming that the square has unit side .In fact, there are exactly two such ellipses; one is oriented horizontally and the other-vertically; see the left panel of Figure 5.These two ellipses beat any circle and any pair of lines see the table in Figure 5 ; they provide the global minimum of the objective function 7.3 .
The fact that the best-fitting conic to a mass distributed in a square is an ellipse explains why for large samples generated from the uniform distribution in a square ellipses dominate over hyperbolas Section 6 .
As the rectangle R gets extended horizontally, that is, as L > 1 increases the best-fitting conic changes.We found that for L < L 1 ≈ 1.2, the best-fitting conic is still an ellipse though it gets elongated compared to the one found for the square, when L 1 .But for L > L 1 , the best-fitting conic abruptly changes from an ellipse to a pair of parallel lines.Those two lines are running through the rectangle horizontally; their equations are y 0.25 and y 0.75 See Figure 6 .
This fact might explain why for longer rectangles, such as R 0, 2 × 0, 1 used in Section 6, the percentage of samples for which the best-fitting ellipse exists does not grow much as n increases.Though it remains unclear which type of conics ellipses or hyperbolas dominates for large samples taken from longer rectangles 0, L × 0, 1 with L > L 1 .

Megaspace
Our conclusions can be illustrated by an interesting construction in a multidimensional space "megaspace".It was first used by Malinvaud 28, Chapter 10 and then by Chernov 4, Sections 1.5 and 3.4 .
Recall that given n data points P 1 x 1 , y 1 , . .., P n x n , y n and a model object S a closed subset of R 2 , our objective function F S is defined by 1.1 and for the distances dist P i , S we can write dist P i , S 2 min x i ,y i ∈S 8.1 Thus we can express the objective function F S as follows: Now let us represent the n data points P 1 x 1 , y 1 , . . ., P n x n , y n by one point "megapoint" P in the 2n-dimensional space R 2n with coordinates x 1 , y 1 , . . ., x n , y n .For the given object S, let us also define a multidimensional set "megaset" M S ⊂ R 2n as follows: Thus, the problem of finding the best-fitting object S best reduces to the problem of finding the megapoint P ∈ M M that is closest to the given megapoint P representing the given n points .In other words, we need to project the megapoint P onto the megaset M M , and the footpoint P of the projection would correspond to the best-fitting object S best .
We conclude that the best-fitting object S best ∈ M exists if and only if there exists a megapoint P ∈ M M that is closest to the given megapoint P. It is a simple fact that given a set D ⊂ R d with d ≥ 1, the closest point X ∈ D to a given point X ∈ R d exists for any X ∈ R d if and only if the set D is closed.Thus the existence of the best-fitting object S best requires the megaset M M to be topologically closed.Again we see that the property of closedness is vital for the fitting problem to have a solution for every set of data points P 1 , . . ., P n .all belonging to the megaset M M , converges, as k → ∞, to a megapoint in the usual topology of R 2n .Equivalently, for every i 1, . . ., n the point . We need to show that P ∞ ∈ M M .Now for every k there exists a model object S k ∈ M that contains the n points The sequence {S k } contains a convergent subsequence S k r , that is, such that S k r → S as r → ∞ for some closed set S ⊂ R 2 .The existence of a convergent subsequence can be verified by the diagonal argument as in the proof of the existence theorem in Section 4.
As we assumed that the collection M is closed, it follows that M contains the limit object S, that is, S ∈ M. It is intuitively clear and can be verified by a routine calculus-type argument that S contains all the limit points x . Therefore the limit megapoint P ∞ belongs to the megaset M M .This proves that the latter is closed.

Megaset for Lines
Let M L consist of all lines in R 2 .The corresponding megaset M M L ⊂ R 2n is described in 28, Chapter 10 .A megapoint x 1 , y 1 , . . ., x n , y n belongs to M M L if and only if all the n planar points x 1 , y 1 , . . ., x n , y n belong to one line i.e., they are collinear .This condition can be expressed by C n,3 algebraic relations as follow: for all 1 ≤ i < j < k ≤ n.Each of these relations means that the three points x i , y i , x j , y j , and x k , y k are collinear; cf.6.1 .All of these relations together mean that all the n points x 1 , y 1 , . . ., x n , y n are collinear.Note that M L is specified by n − 2-independent relations; hence it is an n 2dimensional manifold algebraic variety in R 2n .The relations 8.8 are quadratic, so M L is a quadratic surface.It is closed in topological sense; hence the problem of finding the bestfitting line always has a solution.

Megaset for Circles
Let M C consist of all circles in R 2 .The corresponding megaset M M C ⊂ R 2n is described in 4, Section 3.4 , and we briefly repeat it here.A megapoint x 1 , y 1 , . . ., x n , y n belongs to M M C if and only if all the n planar points x 1 , y 1 , . . ., x n , y n belong to one circle we say that they are "cocircular" .In that case all these points satisfy one quadratic equation of a special type as follow: Bx Cy D 0. 8.9 This condition can be expressed by C n,4 algebraic relations as follow:

Megaset for Ellipses
Let M E consist of all ellipses in R 2 .The corresponding megaset M M E ⊂ R 2n can be described in a similar manner as above.A point x 1 , y 1 , . . ., x n , y n belongs in M M E if and only if all the n planar points x 1 , y 1 , . . ., x n , y n belong to one ellipse we say that they are "coelliptical" .In that case all these points satisfy one quadratic equation of a general type as follow: for 1 ≤ i < j < k < m < l < r ≤ n.Each of these relations means that the six points x i , y i , x j , y j , x k , y k , x m , y m , x l , y l , and x r , y r satisfy one quadratic equation of type 8.12 ; that is, they belong to one conic either regular or degenerate .All of these relations together mean that all the n points x 1 , y 1 , . . ., x n , y n satisfy one quadratic equation of type 8.12 ; that is, they all belong to one conic regular or degenerate .Therefore the relations 8.13 describe a much larger megaset M M Conics corresponding to the collection of all conics, regular and degenerate, that is, using our previous notation.The determinant in 8.13 is a polynomial of the eighth degree, and M M Conics is a closed n 5 -dimensional algebraic manifold in R 2n .It is mostly made of two big megasets: M M E and M M H and they both are n 5 -dimensional.Other megasets parabolas and pairs of lines have smaller dimensions and play the role of the boundaries of the bigger megasets M M E and M M H .

Illustrations
The structure of the megaset M M Conics is schematically depicted in Figure 7, where it is shown as the xy plane {z 0} in the 3D space the latter plays the role of the megaspace R 2n .The positive half-plane H {y > 0, z 0} represents the elliptic megaset M M E , and the negative half-plane H − {y < 0, z 0} represents the hyperbolic megaset M M H .The x-axis {y z 0} separating these two half-planes represents the lower-dimensional megasets M M ∪ ∪ M LL in the decomposition 8.14 .True, the real structure of M M Conics is much more complicated, but our simplified picture still shows its most basic features.Now suppose that the projection point Q 1 coincides with some other projection point s .In that case there are at most four distinct projection points.We will not move the point Q 1 , but we will rotate slightly the tangent line to the conic S at the point Q 1 .Denote that tangent line by T .Note that T is perpendicular to the line passing through P 1 and Q 1 .Now let T be another line passing through the point Q 1 and making an arbitrarily small angle with T .It is not hard to show, by elementary geometry, that there exists another nondegenerate conic S passing though the same ≤4 projection points and having the tangent line T at Q 1 .
Since the tangent line T to the conic S at the point Q 1 is not orthogonal to the line dist P i , S 2 F S , A.5 as desired.Since the line T can be selected arbitrarily close to T , the new conic S will be arbitrarily close to S.

Figure 1 :
Figure 1: Sequence of lines L n and the limit line L.

3 Figure 2 :
Figure 2: Sequence of circles and the limit line.
j k in the sequence S m and a compact subset S * j ⊂ D j such that dist H S j k ∩ D j , S * j → 0 as k → ∞.Next, from the subsequence S j k we extract a subsubsequence, call it S j 1 k

Figure 3 :
Figure 3: Sequences of ellipses converging to various limit objects.

Figure 4 :
Figure 4: Sequences of hyperbolas converging to various limit objects.

Figure 5 :
Figure 5: The best fit to a uniform distribution in a square.

Figure 6 :
Figure 6: The best fit to a uniform distribution in a rectangle.

Theorem 8 . 1
closedness of megasets .If the model collection M is closed (in the sense of W convergence), then the megaset M M is closed in the natural topology of R 2n .Clearly, this theorem provides an alternative proof of the existence of the best-fitting object, provided that the model collection M is closed.Proof.Suppose a sequence of megapoints

8 . 10 for 1 ≤ 8 . 11 A
i < j < k < m ≤ n.Note that 8.10 include megapoints satisfying 8.8 ; hence they actually describe the unionM M C ∪ M M L .The determinant in 8.10 is a polynomial of degree four, andM M C ∪ M M L is an n 3 -dimensional algebraic variety manifold in R 2n defined by quartic polynomial equations.Note that the dimension of M M C ∪ M M L is one higher than that of M M L , that is, dim M M C ∪ M M L dim M M L 1.closer examination shows that M M L plays the role of the boundary of M M C , that is, M M C terminates on M M L .The megaset M M C is not closed, but if we add its boundary M M L to it, it will become closed.

Table 2 :
Percentage % of randomly generated samples of n points for which the best-fitting ellipse exists.
y 1 , ..., x n , y n ∈ M S ⇐⇒ x i , y i ∈ S ∀i.Next given a collection M of model objects we define a large megaset M M ⊂ R 2n as follows: M M ∪ S∈M M S .Alternatively, it can be defined asx 1 , y 1 , . .., x n , y n ∈ M M ⇐⇒ ∃S ∈ M : x i , y i ∈ S ∀i.8.5The best-fitting object S best minimizes the function F S .Thus, due to 8.4 , S best minimizes the distance from the megapoint P representing the data set P 1 , . . ., P n to the megaset M M representing the collection M.
Cy 2 2Dx 2Ey F 0.8.12This equation actually means that the points belong to one conic either regular or degenerate .This condition can be expressed by C n,6 algebraic relations as follow: − x j y jx i − x k y i − y k x 2 i − x 2 k y 2 i − y 2 k x i y i − x k y k x i − x m y i − y m x 2 i − x 2 m y 2 i − y 2 m x i y i − x m y m x i − x l y i − y l x 2 we easily havedist P 1 , S < dist P 1 , Q 1 dist P 1 , S .A.3At the same time we have dist P i , S ≤ dist P i , Q i dist P i , S for every i 2, . . ., 5.