Fitting geometric shapes to observed data points (images) is a popular task in computer vision and modern statistics (errors-in-variables regression). We investigate the problem of existence of the best fit using geometric and probabilistic approaches.
1. Introduction
In many areas of human practice, one needs to approximate a set of points P1,…,Pn∈ℝd (representing experimental data or observations) by a simple geometric shape, such as a line, plane, circular arc, elliptic arc, and spherical surface. This problem is known as fitting a model object (line, circle, sphere, etc.) to observed data points.
The best fit is achieved when the geometric distances from the given points to the model object are minimized, in the least squares sense. Finding the best fit reduces to the minimization of the objective function
(1.1)ℱ(S)=∑i=1n[dist(Pi,S)]2,
where S⊂ℝd denotes the model object (line, circle, sphere, etc.).
While other fitting criteria are used as well, it is the minimum of (1.1) that is universally recognized as the most desirable solution of the fitting problem. It has been adopted as a standard in industrial applications [1, 2]. In statistics, the minimum of (1.1) corresponds to the maximum likelihood estimate under usual assumptions known as functional model; see [3, 4]. Minimization of (1.1) is also called orthogonal distance regression (ODR).
Geometric fitting in the sense of minimizing (1.1) has a long history. First publications on fitting linear objects (lines, planes, etc.) to given points in 2D and 3D date back to the late nineteenth century [5]. The problem of fitting lines and planes was solved analytically in the 1870s [6], and the statistical properties of the resulting estimates were studies throughout the twenties century [7–9], the most important discoveries being made in the 1970s [3, 10, 11]. These studies gave rise to a new branch of mathematical statistics now known as errors-in-variable (EIV) regression analysis [3, 4].
Since the 1950s, fitting circles and spheres to data points became popular in archaeology, industry, and so forth, [12, 13]. In the 1970s, researchers started fitting ellipses and hyperbolas to data points [14–17].
The interest to the problem of geometric fitting surged in the 1990s when it became an agenda issue for the rapidly growing computer science community. Indeed, fitting simple contours to digitized images is one of the basic tasks in pattern recognition and computer vision. In most cases, those contours are lines, circles, ellipses and other conic sections (called simply conics, for brevity) in 2D, and planes, spheres, ellipsoids in 3D. More complicated curves and surfaces are used occasionally too [1, 18], but it is more common to divide complex images into smaller segments and approximate each of those by a line or by a circular arc. This way one can approximate a complex contour by a polygonal line or a circular spline (see, e.g., [19, 20]).
Most publications on the geometric fitting problem are devoted to analytic solutions (which are only possible for lines and planes), or practical algorithms for finding the best-fitting object, or statistical properties of the resulting estimates. Very rarely one addresses fundamental issues such as the existence and uniqueness of the best fit. If these issues do come up, one either assumes that the best fit exists and is unique, or just points out examples to the contrary without deep investigation.
In this paper we address the issue of existence of the best fit in a general and comprehensive manner. The issue of uniqueness will be treated in a separate paper. These issues turn out to be quite nontrivial and lead to unexpected conclusions. As a glimpse of our results, we provide Table 1 summarizing the state of affairs in the problem of fitting 2D objects (here Yes means the best-fitting object exists or is unique in all respective cases; No means the existence/uniqueness fails in some of the respective cases).
Objects
Existence
Existence
Uniqueness
in all cases
in typical cases
Lines
Yes
Yes
No
Circles
No
Yes
No
Ellipses
No
No
No
All conics
No
Yes
No
We see that the existence and uniqueness of the best-fitting object cannot be just taken for granted. Actually 2/3 of the answers in Table 1 are negative. The uniqueness can never be guaranteed. The existence is guaranteed only for lines. In typical cases (i.e., for typical sets of data points, in probabilistic sense; see precise definition in Section 6), the best-fitting line and circle do exist, but the best-fitting ellipse does not.
The existence and uniqueness of the best fit are not only of theoretical interest but also practically relevant. For example, knowing under what conditions the problem does not have a solution might help us understand why the computer algorithm keeps diverging, or returns nonsense, or crashes altogether. While the cases where the best fit does not theoretically exist may be exceptional, nearby cases may be practically hard to handle, as the best-fitting object may be extremely difficult to find.
The nonuniqueness has its practical implications too. It means that the best-fitting object may not be stable under slight perturbations of the data points. An example is described by Nievergelt [21]; he presented a set of n=4 points that can be fitted by three different circles equally well. Then by arbitrarily small changes in the coordinates of the points one can make any of these three circles fit the points a bit better than the other two circles, thus the best-fitting circle will change abruptly. See also a similar example in [4, Section 2.2].
Here we develop a general approach to the studies of existence of the best fit. Our motivation primarily comes from popular applications where one fits lines, circles, ellipses, and other conics, but our methods and ideas can be applied to much more general models. Our approach works in any dimension; that is, we can treat data points P1,…,Pn∈ℝd and model objects S⊂ℝd for any d≥2, but for the sake of notational simplicity and ease of illustrations we mostly restrict the exposition to the planar case d=2.
2. W-Convergence and the Induced Topology
A crucial notion in our analysis is that of convergence (of model objects).
Motivating Example
Consider a sequence of lines Ln={y=x/n}. They all pass through the origin (0,0) and their slopes 1/n decrease as n grows (Figure 1). Naturally, we would consider this sequence as convergent; it converges to the x-axis, that is, to the line L={y=0}, as n→∞.
Sequence of lines Ln and the limit line L.
However, it is not easy to define convergence so that the above sequence of lines would satisfy it. For example, we may try to use Hausdorff distance to measure how close two objects S1andS2⊂ℝ2 are. It is defined by
(2.1)distH(S1,S2)=max{supP1∈S1dist(P1,S2),supP2∈S2dist(P2,S1)}.
If the two sets S1andS2 are closed and the Hausdorff distance between them is zero, that is, distH(S1,S2)=0, then they coincide S1=S2. If the Hausdorff distance is small, the two sets S1andS2 nearly coincide with each other. So the Hausdorff distance seems appropriate for our purposes.
But it turns out that the Hausdorff distance between the lines Ln and L in our example is infinite, that is, distH(Ln,L)=∞ for every n. Thus the Hausdorff distance cannot be used to characterize convergence of lines as we would like to see it.
Windows
So why do we think that the line Ln converges to the line L, despite an infinite Hausdorff distance between them? It is because we do not really see an infinite line, we only “see’’ objects in a certain finite area, like in Figure 1. Suppose we see objects in some rectangle
(2.2)R={-A≤x≤A,-B≤y≤B},
which for the moment will play the role of our “window’’ through which we look at the plane. Then we see segments of our lines within R; that is, we see intersections Ln∩R and L∩R. Now clearly the segment Ln∩R gets closer to L∩R, as n grows, and in the limit n→∞ they become identical. This is why we see the lines Ln converging to L. We see this convergence no matter how large the window R is. Note that the Hausdorff distance between Ln∩R and L∩R indeed converges to zero: distH(Ln∩R,L∩R)→0 as n→∞.
Window-Restricted Hausdorff Distance
Taking cue from the above example, we define the Hausdorff distance between sets S1 and S2within a finite window R as follows:
(2.3)distH(S1,S2;R)=max{supP∈S1∩Rdist(P,S2),supQ∈S2∩Rdist(Q,S1)}.
The formula (2.3) applies whenever both sets, S1 and S2, intersect the window R. If only one set, say S1, intersects R, we modify (2.3) as follows:
(2.4)distH(S1,S2;R)=supP∈S1∩Rdist(P,S2).
A similar modification is used if only S2 intersects R. If neither set intersects the window R, we simply set distH(S1,S2;R)=0 (because we “see’’ two empty sets, which are not distinguishable).
We now define our notion of convergence for a sequence of sets.
Definition 2.1.
Let Sn⊂ℝ2 be some sets and S⊂ℝ2 another set. We say that the sequence Snconverges to S if for any finite window R we have
(2.5)distH(Sn,S;R)→0asn→∞.
In this definition the use of finite windows is crucial; hence we will call the resulting notion Window convergence, or W convergence, for short. According to this definition, the sequence of lines Ln in the above example indeed converges to the limit line L.
For finite-size (bounded) objects, like circles or ellipses, the W-convergence is equivalent to the convergence with respect to the Hausdorff distance. However for unbounded objects, such as lines and hyperbolas, we have to use our window-restricted Hausdorff distance and formula (2.5).
Our definition of convergence (i.e., W-convergence) is intuitively clear, but some complications may arise if it is used too widely. For example, if the limit object S is not necessarily closed, then the same sequence {Sn} may have more than one limit S (though the closure S¯ of every limit set S will be the same). Thus, from now on, to avoid pathological situations, we will assume that all our sets S⊂ℝ2 are closed. All the standard model objects—lines, circles, ellipses, hyperbolas—are closed.
Convergence induces topology in the collection of all closed sets S⊂ℝ2, which we denote by 𝕏. A set Y⊂𝕏 is closed if for any sequence of sets Sn∈Y converging to a limit set S the limit set also belongs to Y, that is, S∈Y. A set U⊂𝕏 is open if its complement 𝕏∖U is closed.
The above topology on 𝕏 is metrizable. This means we can define a metric on 𝕏, that is, a distance between closed sets in ℝ2, in such a way that the W-convergence Sn→S means exactly that the distance between Sn and S goes down to zero. Such a distance can be defined in many ways; here is one of them.
Definition 2.2.
The W distance (or Window distance) between two closed sets S1andS2⊂ℝ2 is defined as follows:
(2.6)distW(S1,S2)=∑k=1∞2-kdistH(S1,S2;Rk),
where Rk is a square window of size 2k×2k, that is, Rk={|x|≤k,|y|≤k}.
In this formula, we use a growing sequence of nested windows and the Hausdorff distances between S1 and S2 within those windows balanced by the factors 2-k. In the formula (2.6), the first nonzero term corresponds to the smallest window Rk that intersects at least one of the two sets, S1 or S2. The sum in (2.6) is always finite. Indeed, let us suppose, for simplicity, that both S1 and S2 intersect each window Rk. Then (since the distance between any two points in Rk is at most 22k) the above sum is bounded by 22∑k=1∞k2-k<6.
3. Continuity of Objective Function
Recall finding the best-fitting object S for a given set of points P1,…,Pn∈ℝ2 consists of minimization of the objective function (1.1).
Let 𝕄 denote the collection of model objects. We put no restrictions on 𝕄 other than all S∈𝕄 are assumed to be closed sets. The collection 𝕄 is then a subset of the space 𝕏 of all closed subsets S⊂ℝ2. The topology and metric on 𝕏 defined above now induce a topology and metric on 𝕄.
Redundancy Principle
If an object S'∈𝕄 is a subset of another object S∈𝕄, that is, S′⊂S, then for any point P we have dist(P,S)≤dist(P,S′); thus S′ cannot fit any set of data points better than S does. So for the purpose of minimizing ℱ, that is, finding the best-fitting object, we may ignore all model objects that are proper subsets of other model objects (they are redundant). This may reduce the collection 𝕄 somewhat. Such a reduction is not necessary, it is just a matter of convenience, and we will occasionally apply it below.
Conversely, there is no harm in considering any subset S′⊂S of an object S∈𝕄 as a (smaller) object, too. Indeed, if S′ provides a best fit (i.e., minimizes the objective function ℱ), then so does S, because ℱ(S)≤ℱ(S′). Hence including S′ into the collection 𝕄 will not really be an extension of 𝕄, its inclusion will not change the best fit.
Theorem 3.1 (continuity of <inline-formula><mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" id="M133"><mml:mrow><mml:mi>ℱ</mml:mi></mml:mrow></mml:math></inline-formula>).
For any given points P1,…,Pn and any collection 𝕄 of model objects the function ℱ defined by (1.1) that is continuous on 𝕄. This means that if a sequence of objects Sm∈𝕄 converges (i.e., W converges) to another object S∈𝕄, then ℱ(Sm)→ℱ(S).
Proof.
Since ℱ(S) is the sum of squares of distances dist(Pi,S) to individual points Pi, see (1.1), it is enough to verify that the distance dist(P,S) is a continuous function of S for any given point P.
Suppose P∈ℝ2 and a sequence of closed sets Sm W converging to a closed set S. We denote by Q∈S the point in S closest to P, that is, such that dist(P,Q)=dist(P,S); such a point Q exists because S is closed. Denote by D the disk centered on P of radius 1+dist(P,Q); it contains Q. Let R be a window containing the disk D.
Since R contains Q, it intersects with S, that is, R∩S≠∅. This guarantees that distH(Sm,S;R)→0. Thus, there are points Qm∈Sm such that Qm→Q. Since dist(P,Sm)≤dist(P,Qm), we conclude that the upper limit of the sequence dist(P,Sm) does not exceed dist(P,S), that is,
(3.1)limsupdist(P,Sm)≤dist(P,S).
On the other hand, we will show that the lower limit of the sequence dist(P,Sm) cannot be smaller than dist(P,S), that is,
(3.2)liminfdist(P,Sm)≥dist(P,S).
The estimates (3.1) and (3.2) together imply that dist(P,Sm)→dist(P,S), as desired, and hence the distance function dist(P,S) will be continuous on 𝕄. It remains to prove (3.2).
To prove (3.2) we assume that it is false. Then there is a subsequence Smk in our sequence of sets Sm such that
(3.3)limk→∞dist(P,Smk)=liminfdist(P,Sm)<dist(P,S).
Denote by Qm∈Sm the point in Sm closest to P, that is, such that dist(P,Qm)=dist(P,Sm). Then we have
(3.4)limk→∞dist(P,Qmk)=limk→∞dist(P,Smk)<dist(P,S)=dist(P,Q).
Since the points Qmk are closer to P than the point Q is, we have Qmk∈D⊂R. Recall that distH(Sm,S;R)→0, hence
(3.5)dist(Qmk,S)→0ask→∞.
Denote by Hmk∈S the point in S closest to Qmk, that is, such that dist(Qmk,Hmk)=dist(Qmk,S). Now we have, by triangle inequality,
(3.6)dist(P,S)≤dist(P,Hmk)≤dist(P,Qmk)+dist(Qmk,Hmk)=dist(P,Qmk)+dist(Qmk,S).
The limit of the first term on the right hand side of (3.6) is <dist(P,S) by (3.4), and the limit of the second term is zero by (3.5). This implies dist(P,S)<dist(P,S), which is absurd. The contradiction proves (3.2). And the proof of (3.2) completes the proof of the theorem.
4. Existence of the Best Fit
The best-fitting object Sbest∈𝕄 corresponds to the (global) minimum of the objective function ℱ, that is, Sbest=argminS∈𝕄ℱ(S). The function ℱ defined by (1.1) cannot be negative; thus, it always has an infimum
(4.1)ℱ0=infS∈𝕄ℱ(S).
This means that one cannot find objects S∈𝕄 such that ℱ(S)<ℱ0, but one can find objects S∈𝕄 such that ℱ(S) is arbitrarily close to ℱ0. More precisely, there is a sequence of objects Sm such that ℱ(Sm)>ℱ0 for each m and ℱ(Sm)→ℱ0 as m→∞.
In practical terms, one usually runs a computer algorithm that executes a certain iterative procedure. It produces a sequence of objects Sm (here m denotes the iteration number) such that ℱ(Sm)<ℱ(Sm-1); that is, the quality of approximations improves at every step. If the procedure is successful, the value ℱ(Sm) converges to the minimal possible value, ℱ0, and the sequence of objects Sm converges (i.e., W-converges) to some limit object S0. Then the continuity of the objective function ℱ (proven earlier) guarantees that ℱ(S0)=ℱ0; that is, S0 indeed provides the global minimum of the objective function, so it is the best-fitting object S0=Sbest.
A problem arises if the limit object S0 does not belong to the given collection 𝕄; hence it is not admissible. Then we end up with a sequence of objects Sm, each of which fits (approximates) the given points better than the previous one, but not as good as the next one. None of them would qualify as the best fit, and in fact the best fit would not exist, so the fitting problem would have no solution.
Illustrative Example
Suppose that our model objects are circles, and our given points are P1=(-1,0), P2=(0,0) and P3=(1,0). Then the sequence of circles Sm defined by x2+(y-m)2=m2 will fit the points progressively better (tighter) as m grows, so that ℱ(Sm)→0 as m→∞ (Figure 2). On the other hand, no circle can pass through three collinear points; hence no circle S satisfies ℱ(S)=0. Thus the circle-fitting problem has no solution.
Sequence of circles and the limit line.
The above sequence of circles Sm converges (in our terms, W-converges) to the x axis, which is a line, so it is natural to declare that line “the best fit.’’ This may not be satisfactory in some practical applications, where one really needs to produce an estimate of the circle’s center and radius. But if we want to present the best-fitting object here, it is clearly and undeniably the line y=0.
Thus, in order to guarantee the existence of the best-fitting object in all cases, we need to include in our collection 𝕄 all objects that can be obtained as limits of sequences of objects from 𝕄. Such “limit objects’’ are called limit points of 𝕄, in the language of topology.
Definition 4.1.
A collection 𝕄 which already contains all its “limit points’’ is said to be closed. If the collection 𝕄 is not closed, then an extended collection 𝕄¯ that includes all the limit points of 𝕄 is called the closure of 𝕄.
For example, the collection 𝕄L of all lines in ℝ2 is closed, as a sequence of lines can only converge to a line. The collection 𝕄C of all circles in ℝ2 is not closed, as a sequence of circles may converge to a line. The closure of the collection of circles 𝕄C includes all circles and all lines, that is,
(4.2)𝕄¯C=𝕄C∪𝕄L.
(Strictly speaking, a sequence of circles may also converge to a single point, see Section 5, so singletons need to be included too.)
We see that the collection 𝕄 of model objects must be closed if we want the best-fitting object to exist in all cases. If 𝕄 is not closed, we have to extend it by adding all its limit points and thus make it closed.
Theorem 4.2 (existence of the best fit).
Suppose that the given collection 𝕄 of model objects is closed. Then for any given points P1,…,Pn there exists the best-fitting object Sbest∈𝕄, that is, the objective function ℱ attains its global minimum on 𝕄.
Proof.
The key ingredients of our proof will be the continuity of the objective function ℱ and the compactness of a restricted domain of that function. We recall that a continuous real-valued function on a compact set always takes its maximum value and its minimum value on that set. In metric spaces (like our 𝕄), a subset 𝕄0⊂𝕄 is compact if every sequence of its elements Sm∈𝕄0 has a subsequence Smk that converges to another element S∈𝕄0, that is, Smk→S as k→∞.
We recall that our objective function ℱ is continuous, and its domain 𝕄 is now assumed to be closed. If it was compact, the above general fact would guarantee that ℱ had a global minimum, as desired. But in most practical settings 𝕄 is not compact. Indeed, if it was compact, the function ℱ would not only have a minimum, but also a maximum. And this is usually impossible; one can usually find model objects arbitrarily far from the given points, thus making the distances from the points to the object arbitrarily large. Thus we need to find a smaller (restricted) subcollection 𝕄0⊂𝕄 which will be compact, and then we will apply the above general fact.
Let Dr={x2+y2≤r2} denote the disk of radius r centered on the origin (0,0). We can find a large enough disk Dr that satisfies two conditions: (i) Dr covers all the given points P1,…,Pn and (ii) Dr intersects at least one object S0∈𝕄, that is, D∩S0≠∅. The distances from the given points to S0 cannot exceed the diameter of Dr, which is 2r, hence ℱ(S0)≤(2r)2n.
Now we define our subcollection 𝕄0⊂𝕄; it consists of all model objects S∈𝕄 that intersect the larger disk D3r of radius 3r. Objects that lie entirely outside D3r are not included in 𝕄0. Note that the subcollection 𝕄0 contains at least one object (S0 mentioned above); hence 𝕄0 is not empty.
Recall that all our given points P1,…,Pn lie in Dr. They are separated from the region outside the larger disk D3r by a “no man’s land’’—the ring D3r∖Dr, which is 2r wide. Thus the distances from the given points to any object S which was not included in 𝕄0 are greater than 2r; hence for such objects we have ℱ(S)>(2r)2n. So objects not included in 𝕄0 cannot fit our points better than S0 does. Thus they can be ignored in the process of minimization of ℱ. More precisely, if we find the best-fitting object Sbest within the subcollection 𝕄0, then for any other object S∈𝕄∖𝕄0 we will have
(4.3)ℱ(S)>(2r)2n≥ℱ(S0)≥ℱ(Sbest),
which shows that Sbest will be also the best-fitting object within the entire collection 𝕄.
It remains to verify that the subcollection 𝕄0 is compact; that is, every sequence of objects Sm∈𝕄0 has a subsequence converging to another object S*∈𝕄0. We will use the following standard fact. In a compact metric space any sequence of compact subsets has a convergent subsequence with respect to the Hausdorff metric.
Now let j=[3r]+1 be the smallest integer greater than 3r. Recall that all the objects in 𝕄0 are required to intersect D3r; thus, they all intersect Dj as well. The sets Sm∩Dj are compact. By the above general fact, there is a subsequence Sk(j) in the sequence Sm and a compact subset Sj*⊂Dj such that distH(Sk(j)∩Dj,Sj*)→0 as k→∞. Next, from the subsequence Sk(j) we extract a subsubsequence, call it Sk(j+1) that converges in the larger disk Dj+1; that is, such that distH(Sk(j+1)∩Dj+1,Sj+1*)→0 as k→∞ for some compact subset Sj+1*⊂Dj+1. Since distH(Sk(j+1)∩Dj,Sj*)→0, we see that Sj+1*∩Dj=Sj*; that is, the limit sets Sj* and Sj+1* “agree’’ within Dj.
Then we continue this procedure inductively for the progressively larger disks Dj+2,Dj+3,…. In the end we use standard Cantor’s diagonal argument to construct a single subsequence Smk such that for every i≥j we have distH(Smk∩Di,Si*)→0 as k→∞, and the limiting subsets Si*⊂Di “agree’’ in the sense Si+1*∩Di=Si* for every i≥j. Then it follows that the sequence of objects Smk converges (i.e., W converges) to the closed set S*=∪i≥jSi*. The limit set S* must belong to our collection 𝕄 because that collection was assumed to be closed. Lastly, S* intersects the disk D3r, so it also belongs to the subcollection 𝕄0. Our proof is now complete.
5. Some Popular ModelsLines and Planes
The most basic (and oldest) fitting application is fitting lines in 2D. The model collection 𝕄L consists of all lines L⊂ℝ2. It is easy to see that a sequence of lines Lm can only converge to another line; hence, the collection 𝕄L of lines is closed. This guarantees that the fitting problem always has a solution.
Similarly, the collection of lines in 3D is closed. The collection of planes in 3D is closed, too. Thus the problems of fitting lines and planes always have a solution. The same is true for the problem of fitting k-dimensional planes in a d-dimensional space ℝd for any 1≤k<d.
Circles and Spheres
A less-trivial task is fitting circles in 2D. Now the model collection 𝕄C consists of all circles C⊂ℝ2. A sequence of circles Cm can converge to an object of three possible types: a circle C0⊂ℝ2, a line L0⊂ℝ2, or a single point (singleton) P0∈ℝ2. Singletons can be regarded as degenerate circles (whose radius is zero), or they can be ignored based on the redundancy principle introduced in Section 3. But the lines are objects of a different type; thus, the collection 𝕄C of circles is not closed.
As a result, the circle fitting problem does not always have a solution; a simple example was given in Section 4. In order to guarantee the existence of the best-fitting object, one needs to extend the given collection 𝕄C by adding all its “limit points,’’ in this case-lines. The extended collection will include all circles and all lines, that is,
(5.1)𝕄¯C=𝕄C∪𝕄L.
Now the (extended) circle fitting problem always has a solution; for any data points P1,…,Pn there is a best-fitting object Sbest∈𝕄¯C that minimizes the sum of squares of the distances to P1,…,Pn. One has to keep in mind, though, that the best-fitting object may be a line, rather than a circle.
The problem of fitting circles to data points has been around since the 1950’s [13]. Occasional nonexistence of the best-fitting circle has been first noticed empirically; see, for example, Berman and Culpin [22]. The first theoretical analysis of the non-existence phenomenon was done by Nievergelt [21] who traced it to the noncompactness of the underlying model space and concluded that lines needed to be included in the model space to guarantee the existence of the best fit. He actually studied this phenomenon in many dimensions, so he dealt with hyperspheres and hyperplanes. He called hyperplanes “generalized hyperspheres.’’ Similar analysis and conclusions, for circles and lines, were published later, independently, by Chernov and Lesort [23] and by Zelniker and Clarkson [24]; see also a recent book [4].
Ellipses
A popular task in computer vision is fitting ellipses. Now the model collection 𝕄E consists of all ellipses E⊂ℝ2. A sequence of ellipses Em may converge to an object of many possible types: an ellipse, a parabola, a line, a pair of parallel lines, a segment of a line, a half-line (a ray); see Figure 3.
Sequences of ellipses converging to various limit objects.
Line segments can be regarded as degenerate ellipses whose minor axis is zero. Alternatively, line segments and half-lines can be ignored based on the redundancy principle (Section 3). Thus we have lines, pairs of parallel lines, and parabolas as limit objects of different types.
We see that the collection 𝕄E of ellipses is not closed. As a result, the ellipse-fitting problem does not always have a solution. For example, this happens when given points that are placed on a line or on a parabola. (Actually, there are many more examples; see Section 6.)
In order to guarantee the existence of the best-fitting object, one needs to extend the collection 𝕄E of ellipses by adding all its “limit points,’’ in this case—lines, pairs of parallel lines, and parabolas. We denote that extended collection by 𝕄¯E that
(5.2)𝕄¯E=𝕄E∪𝕄L∪𝕄∥∪𝕄∪,
where 𝕄∥ denotes the collection of pairs of parallel lines, and 𝕄∪ the collection of parabolas (the symbol ∪ resembles a parabola).
Now the (extended) ellipse-fitting problem always has a solution, for any data points P1,…,Pn there will be a best-fitting object Sbest∈𝕄¯E that minimizes the sum of squares of the distances to P1,…,Pn. One has to keep in mind, though, that the best-fitting object may be a line, or a pair of parallel lines, or a parabola, rather than an ellipse.
The ellipse fitting problem has been around since the 1970s. The need to deal with limiting cases was first notice by Bookstein [16] who wrote “The fitting of a parabola is a limiting case, exactly transitional between ellipse and hyperbola. As the center of ellipse moves off toward infinity while its major axis and the curvature of one end are held constant….’’ A first theoretical analysis on the non-existence of the best ellipse was done by Nievergelt in [25] who traced it to the noncompactness of the underlying model space and concluded that parabolas needed to be included in the model space to guarantee the existence of the best fit.
Conics
We now turn to a more general task of fitting quadratic curves (also known as conic sections, or simply conics). Now the model collection 𝕄Q consists of all nondegenerate quadratic curves, by which we mean ellipses, parabolas, and hyperbolas. A sequence of conics may converge to an object of many possible types: a conic, a line, a line segment, a ray, two opposite rays, and a pair of parallel lines, and a pair of intersecting lines; see Figures 3 and 4.
Sequences of hyperbolas converging to various limit objects.
Every line segment, ray, and two opposite rays are a part (subset) of a full line, so these objects can be ignored based on the redundancy principle (Section 3). But pairs of parallel lines and pairs of intersecting lines constitute limit objects of new types. We see that the collection 𝕄Q of quadratic curves (conics) is not closed. As a result, the conic fitting problem does not always have a solution.
In order to guarantee the existence of the best-fitting object, one needs to extend the collection 𝕄Q of quadratic curves by adding all its “limit points’’—pairs of lines (which may be intersecting, parallel, or coincident)
(5.3)𝕄¯Q=𝕄Q∪𝕄LL,
where 𝕄LL denotes the collection of pairs of lines.
Now the (extended) conic fitting problem always has a solution; for any data points P1,…,Pn there is a best-fitting object Sbest∈𝕄¯Q that minimizes the sum of squares of the distances to P1,…,Pn. One has to keep in mind, though, that the best-fitting object may be a pair of lines, rather than a genuine conic.
6. Sufficient and Deficient Models
We see that if the collection 𝕄 of adopted model objects is not closed, then in order to guarantee the existence of the best fit we have to close it up, that is, add to 𝕄 all objects that are obtained as limits of existing objects. The new, extended collection 𝕄¯ will be closed and will always provide the best fit. Here we discuss practical implications of extending 𝕄.
Definition 6.1.
The original objects S∈𝕄 are called primary objects and the objects that have been added, that is, S∈𝕄¯∖𝕄 are called secondary objects.
The above extension of 𝕄 by secondary objects, though necessary, may not be completely harmless. Suppose again that one needs to fit circles to observed points. Since the collection of circles 𝕄C is not closed, one has to add lines to it. That is, one really has to operate with the extended collection 𝕄¯C=𝕄C∪𝕄L in which circles are primary objects and lines are secondary objects. The best-fitting object to a given set of points will then belong to 𝕄¯C; hence occasionally it will be a line, rather than a circle.
In practical applications, though, a line may not be totally acceptable as the best-fitting object. For example, if one needs to produce an estimate of the circle’s radius and center, a line will be of little help; it has no radius or center. Thus, when a secondary object happens to be the best fit to the given data points, various unwanted complications may arise.
Next we investigate how frequently such unwanted events occur.
In most applications points are obtained (measured or observed) with some random noise. The noise has a probability distribution, which in statistical studies is usually assumed to be normal [3, 4]. For our purposes it is enough to just assume that given points P1,…,Pn have an absolutely continuous distribution:
Assumption 6.2.
The points P1,…,Pn are obtained randomly with a probability distribution that has a joint probability density ρ(x1,y1,…,xn,yn)>0.
Let us first see how frequently the best-fitting circle fails to exist under this assumption. We begin with the simplest case of n=3 points. If they are not collinear, then they can be interpolated by a circle. If they are collinear (and distinct), there is no interpolating circle, so the best fit is achieved by a line. The three points (xi,yi), 1≤i≤3 are collinear if and only if
(6.1)(x1-x2)(y1-y3)=(x1-x3)(y1-y2).
This equation defines a hypersurface in ℝ6. Under the above assumption, the probability of that hypersurface is zero; that is, the collinearity occurs with probability zero. In simple practical terms, it is “impossible,’’ it “never happens.’’ If we generate three data points by using a computer random number generator, we will practically never see collinear points. All our practical experience tells us; the best-fitting circle always exists. Perhaps for this reason the circle-fitting problem is usually studied without including lines in the collection of model objects.
In the case of n>3 data points, the best-fitting circle fails to exist when the points are collinear, but there are other instances too. For example, let n=4 points be at (0,1), (0,-1), (A,0), and (-B,0) for some large A,B≫1. Then it is not hard to check, by direct inspection, that the best-fitting circle fails to exist, and the best fit is achieved by the line y=0.
It is not easy to describe all sets of points for which the best-fitting circle fails to exist, but such a description exists—it is given in [4] (see Theorem 8 on page 68 there). We skip technical details, only state the final conclusion; for every n>3, the best-fitting circle exists unless the (2n)-vector (x1,y1,…,xn,yn) belongs to a certain algebraic submanifold in ℝ2n. So, under our assumption, the best-fitting circle exists with probability one.
In other words, no matter how large or small the data set is, the best-fitting circle will exist with probability one, so practically we never have to resort to secondary objects (lines); that is, we never have to worry about existence. The model of circles is adequate and sufficient; it does not really require an extension.
Definition 6.3.
We say that a collection 𝕄 of model objects is sufficient (for fitting purposes) if the best-fitting object Sbest exists with probability one under the above assumption. Otherwise the model collection will be called deficient.
We note that under our assumption an event occurs with probability zero if and only if it occurs on a subset A⊂ℝ2n of Lebesgue measure zero, that is, Leb(A)=0. Therefore, our notion of sufficiency does not depend on the choice of the probability density ρ.
As we have just seen, the collection of circles is sufficient for fitting purposes. Next we examine the model collection 𝕄E of ellipses E⊂ℝ2. As we know, this collection is not closed; its closure must include all lines, pairs of parallel lines, and parabolas.
Ellipses
Let us see how frequently the best-fitting ellipse fails to exist. It is particularly easy to deal with n=5 data points. For any set of distinct 5 points in a general linear position (which means that no three points are collinear), there exists a unique quadratic curve (conic) that passes through all of them (i.e., interpolates them). That conic may be an ellipse, or a parabola, or a hyperbola. If 5 points are not in generic linear position (i.e., at least three of them are collinear), then they can be interpolated by a degenerate conic (a pair of lines).
If the interpolating conic is an ellipse, then obviously that ellipse is the best fit (the objective function takes it absolute minimum value, zero). What if the interpolating conic is a parabola or a pair of parallel lines? Then it is a secondary object in the extended model (5.2), and we have an unwanted event; a secondary object provides the best fit. We remark, however, that this event occurs with probability zero, so it is not a real concern. But there are other ways in which unwanted events occur, see below.
Suppose that the interpolating conic is a hyperbola or a pair of intersecting lines. Now the situation is less clear, as such an interpolating conic would not belong to the extended collection 𝕄¯E; see (5.2). Is it possible then that the best-fitting object Sbest∈𝕄¯E is an ellipse? The answer is “no’’, and our argument is based on the following theorem that is interesting itself:
Theorem 6.4 (no local minima for <inline-formula><mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" id="M438"><mml:mi>n</mml:mi><mml:mo>=</mml:mo><mml:mn mathvariant="normal">5</mml:mn></mml:math></inline-formula>).
For any set of five points the objective function ℱ has no local minima on conics. In other words, for any non interpolating conic S there exist other conics S′, arbitrarily close to S, which fit our data points better than S does, that is, ℱ(S′)<ℱ(S).
The proof is given in the appendix, and here we answer the previous question. Suppose n=5 points are interpolated by a hyperbola or a pair of intersecting lines. If there were a best-fitting ellipse Ebest∈𝕄¯E, then no other ellipse could provide a better fit; that is, for any other ellipse E∈𝕄E we would have ℱ(E)≥ℱ(Ebest), which contradicts the above theorem. Thus the best-fitting object Sbest∈𝕄¯E is not an ellipse but a secondary object (a parabola, or a line, or a pair of parallel lines), that is, an unwanted event occurs.
One can easily estimate the probability of the above unwanted event numerically. For a given probability distribution one can generate random samples of n=5 points and for each sample find the interpolating conic (by elementary geometry). Every time that conic happens to be other than an ellipse, an unwanted event occurs; that is, the best fit from within the collection (5.2) is provided by a secondary object.
We have run the above experiment with a standard normal distribution, where each coordinate xi and yi (for i=1,…,5) was generated independently according to a standard normal law 𝒩(0,1). We found that the random points were interpolated by an ellipse with probability 22% and by a hyperbola with probability 78%. (Other conics, including parabolas, never turned up; in fact, it is not hard to prove that they occur with probability zero.) This is a striking result; hyperbolas actually dominate over ellipses! Thus the best-fitting ellipse fails to exist with a probability as high as 78%.
We note that this percentage will remain the same if we generate points (xi,yi) independently according to any other 2D normal distribution. Indeed, any 2D normal distribution can be transformed to a standard 2D normal distribution by a linear map of the plane ℝ2. Now under linear transformations conics are transformed to conics, and their types are preserved (i.e., ellipses are transformed to ellipses, hyperbolas to hyperbolas, etc.). Therefore the type of the interpolating conic cannot change.
In another experiment we sampled points from the unit square [0,1]×[0,1] with a uniform distribution; that is, we selected each coordinate xi and yi independently from the unit interval [0,1]. In this experiment ellipses turned up with probability 28% and hyperbolas with probability 72%, again an overwhelming domination of hyperbolas!
And again, these percentages will remain the same if we generate points (xi,yi) according to any other rectangular (uniform) distribution. Indeed, any rectangle can be transformed to the square [0,1]×[0,1] by a linear map of the plane, and then the same argument as before applies.
We also conducted numerical tests for n>5 points. In these tests we generated points P1,…,Pn, independently according to four predefined distributions: normal with covariance matrix V1=[1001], normal with covariance matrix V2=[4001], uniform in the square [0,1]×[0,1], uniform in the rectangle [0,2]×[0,1]. Table 2 shows the percentage of randomly generated samples of n points for which the best-fitting ellipse exists.
Percentage (%) of randomly generated samples of n points for which the best-fitting ellipse exists.
Distribution
n=5
n=6
n=8
n=10
n=20
n=50
Normal V1
22
23
23
24
27
27
Normal V2
22
22
24
26
27
33
Square [0,1]×[0,1]
28
32
40
46
52
60
Rectangle [0,2]×[0,1]
28
29
28
30
31
32
We see that in most cases the best-fitting ellipse fails to exist with a high probability. Only for the uniform distribution is the square [0,1]×[0,1] the failures decrease as n→∞, and apparently the probability of existence of the best-fitting ellipse grows to 100%. We will explain this in Section 7.
We note that for n≥6 points there is no interpolating conic, so one has to find a global minimum of ℱ via an extensive search over the model space 𝕄Q of conics. This is a very time-consuming procedure and its results are never totally reliable. Thus our estimates, especially for larger n’s, are quite approximate, but we believe that they are accurate to within 5%.
Insufficiency of Ellipses
We conclude that the collection of ellipses is not sufficient for fitting purposes. This means that there is a real chance that for a given set of data points no ellipse could be selected as the best fit to the points; that is, the ellipse-fitting problem would have no solution. More precisely, for any ellipse E there will be another ellipse E' providing a better fit, in the sense ℱ(E')<ℱ(E). If one constructs a sequence of ellipses that fit the given points progressively better and on which the objective function ℱ converges to its infimum, then those ellipses will grow in size and converge to something different than an ellipse (most likely, to a parabola).
In practical terms, one usually runs a computer algorithm that executes an iterative procedure such as Gauss-Newton or Levenberg-Marquardt. It produces a sequence of ellipses Em (here m denotes the iteration number) such that ℱ(Em)<ℱ(Em-1); that is, the quality of approximations improves at every step, but those ellipses would keep growing in size and approach a parabola. This situation commonly occurs in practice [16, 25, 26], and one either accepts a parabola as the best fit or choose an ellipse by an alternative procedure (e.g., by sirect ellipse fit [27]).
We see that whenever the best-fitting ellipse fails to exist, the ellipse-fitting procedure will attempt to move beyond the collection of ellipses, so it will end up on the border of that collection and then return a secondary object (a parabola or a pair of lines). In a sense, the scope of the collection of ellipses is too narrow for fitting purposes. In other words, this collection is deficient, or badly incomplete. Its deficiency indicates that it should be substantially extended for the fitting problem to have a reasonable (adequate) solution with probability one.
Conics
Such an extension is achieved by the collection of all conics (ellipses, hyperbolas, and parabolas) denoted by 𝕄Q; see Section 5. If one searches for the best-fitting object Sbest in the entire collection of conics, then Sbest will exist with probability one. This was confirmed in numerous computer experiments that we have conducted. In fact, the best-fitting object has always been either an ellipse or hyperbola, so parabolas apparently can be excluded. Thus our numerical tests strongly suggest that the collection 𝕄EH=𝕄E∪𝕄H of ellipses, and hyperbolas is sufficient (though we do not have a proof).
7. Generalizations
In the basic formula (1.1) all the n points make equal contributions to ℱ. In some applications, though, one needs to minimize a weighted sum
(7.1)ℱw(S)=∑i=1nwi[dist(Pi,S)]2,
for some fixed wi>0. This problem is known as weighted least squares. All our results on the existence of the best fit apply to it as well.
More generally, given a probability distribution dP(x,y) on ℝ2, one can “fit’’ an object S to it (or approximate it) by minimizing
(7.2)ℱP(S)=∬d2(x,y)dP(x,y),
where d(x,y)=dist[(x,y),S]. Note that (7.1) is a particular version of (7.2) in which dP is a discrete measure with a finite support. In order to guarantee the finiteness of the integral in (7.2) one needs to assume that the distribution dP has finite second moments. After that all our results apply.
The generalization (7.2) is relevant to the numerical tests described in Section 6. Suppose again that n points are selected independently according to a probability distribution dP(x,y) on ℝ2. As n grows, due to the law of large numbers of probability theory that the normalized objective function (1/n)ℱ converges exactly to the integral (7.2).
Now the global minimum of the function ℱP(S) would give us the best-fitting object corresponding to the given probability distribution dP, which would be the best “asymptotic’’ fit to a random sample of size n selected from the distribution dP, as n→∞. This relation helps us explain some experimental results reported in Section 6.
Suppose that dP is a uniform distribution in a rectangle R=[0,L]×[0,1]. Then the integral in (7.2) becomes
(7.3)ℱP(S)=L-1∫0L∫01d2(x,y)dydx.
We have computed it numerically in order to find the best-fitting conic S. The results are given below.
One would naturally expect that the best fit to a mass uniformly distributed in a perfect square would be either a circle or a pair of lines (say, the diagonals of the square).
What we found was totally unexpected; the best-fitting conic is an ellipse! Its center coincides with the center of the square, and its axes are aligned with the sides of the square, but its axes are different; the major axis is 2a=0.9489 and the minor axis is 2b=0.6445 (assuming that the square has unit side). In fact, there are exactly two such ellipses; one is oriented horizontally and the other—vertically; see the left panel of Figure 5. These two ellipses beat any circle and any pair of lines (see the table in Figure 5); they provide the global minimum of the objective function (7.3).
The best fit to a uniform distribution in a square.
The fact that the best-fitting conic to a mass distributed in a square is an ellipse explains why for large samples generated from the uniform distribution in a square ellipses dominate over hyperbolas (Section 6).
As the rectangle R gets extended horizontally, that is, as L>1 increases the best-fitting conic changes. We found that for L<L1≈1.2, the best-fitting conic is still an ellipse (though it gets elongated compared to the one found for the square, when L=1). But for L>L1, the best-fitting conic abruptly changes from an ellipse to a pair of parallel lines. Those two lines are running through the rectangle horizontally; their equations are y=0.25 and y=0.75 (See Figure 6).
The best fit to a uniform distribution in a rectangle.
This fact might explain why for longer rectangles, such as R=[0,2]×[0,1] used in Section 6, the percentage of samples for which the best-fitting ellipse exists does not grow much as n increases. Though it remains unclear which type of conics (ellipses or hyperbolas) dominates for large samples taken from longer rectangles [0,L]×[0,1] with L>L1.
8. Megaspace
Our conclusions can be illustrated by an interesting construction in a multidimensional space “megaspace’’. It was first used by Malinvaud [28, Chapter 10] and then by Chernov [4, Sections 1.5 and 3.4].
Recall that given n data points P1=(x1,y1),…, Pn=(xn,yn) and a model object S (a closed subset of ℝ2), our objective function ℱ(S) is defined by (1.1) and for the distances dist(Pi,S) we can write
(8.1)[dist(Pi,S)]2=min(xi′,yi′)∈S{(xi-xi′)2+(yi-yi′)2}.
Thus we can express the objective function ℱ(S) as follows:
(8.2)ℱ(S)=min{∑i=1n[(xi-xi′)2+(yi-yi′)2];(xi′,yi′)∈S∀i}.
Now let us represent the n data points P1=(x1,y1),…,Pn=(xn,yn) by one point (“megapoint’’) 𝒫 in the 2n-dimensional space ℝ2n with coordinates x1,y1,…,xn,yn. For the given object S, let us also define a multidimensional set “megaset’’ 𝔐S⊂ℝ2n as follows:
(8.3)𝒫'=(x1′,y1′,…,xn′,yn′)∈𝔐S⟺(xi′,yi′)∈S∀i.
Note that ∑i=1n[(xi-xi′)2+(yi-yi′)2] in (8.2) is the square of the distance from 𝒫 to 𝒫'∈𝔐S, in the megaspace ℝ2n. Therefore consider
(8.4)ℱ(S)=min𝒫'∈𝔐S[dist(𝒫,𝒫′)]2=[dist(𝒫,𝔐S)]2.
Next given a collection 𝕄 of model objects we define a large megaset 𝔐(𝕄)⊂ℝ2n as follows: 𝔐(𝕄)=∪S∈𝕄𝔐S. Alternatively, it can be defined as
(8.5)(x1′,y1′,…,xn′,yn′)∈𝔐(𝕄)⟺∃S∈𝕄:(xi′,yi′)∈S∀i.
The best-fitting object Sbest minimizes the function ℱ(S). Thus, due to (8.4), Sbest minimizes the distance from the megapoint 𝒫 representing the data set P1,…,Pn to the megaset 𝔐(𝕄) representing the collection 𝕄.
Thus, the problem of finding the best-fitting object Sbest reduces to the problem of finding the megapoint 𝒫′∈𝔐(𝕄) that is closest to the given megapoint 𝒫 (representing the given n points). In other words, we need to project the megapoint 𝒫 onto the megaset 𝔐(𝕄), and the footpoint 𝒫′ of the projection would correspond to the best-fitting object Sbest.
We conclude that the best-fitting object Sbest∈𝕄 exists if and only if there exists a megapoint 𝒫′∈𝔐(𝕄) that is closest to the given megapoint 𝒫. It is a simple fact that given a set D⊂ℝd with d≥1, the closest point X′∈D to a given point X∈ℝd exists for any X∈ℝd if and only if the set D is closed. Thus the existence of the best-fitting object Sbest requires the megaset 𝔐(𝕄) to be topologically closed. Again we see that the property of closedness is vital for the fitting problem to have a solution for every set of data points P1,…,Pn.
Theorem 8.1 (closedness of megasets).
If the model collection 𝕄 is closed (in the sense of W convergence), then the megaset 𝔐(𝕄) is closed in the natural topology of ℝ2n.
Clearly, this theorem provides an alternative proof of the existence of the best-fitting object, provided that the model collection 𝕄 is closed.
Proof.
Suppose a sequence of megapoints
(8.6)𝒫(k)=(x1(k),y1(k),…,xn(k),yn(k)),
all belonging to the megaset 𝔐(𝕄), converges, as k→∞, to a megapoint
(8.7)𝒫(∞)=(x1(∞),y1(∞),…,xn(∞),yn(∞))
in the usual topology of ℝ2n. Equivalently, for every i=1,…,n the point (xi(k),yi(k)) converges, as k→∞, to the point (xi(∞),yi(∞)). We need to show that 𝒫(∞)∈𝔐(𝕄).
Now for every k there exists a model object S(k)∈𝕄 that contains the n points (x1(k),y1(k)),…,(xn(k),yn(k)). The sequence {S(k)} contains a convergent subsequence Skr, that is, such that Skr→S as r→∞ for some closed set S⊂ℝ2. (The existence of a convergent subsequence can be verified by the diagonal argument as in the proof of the existence theorem in Section 4.)
As we assumed that the collection 𝕄 is closed, it follows that 𝕄 contains the limit object S, that is, S∈𝕄. It is intuitively clear (and can be verified by a routine calculus-type argument) that S contains all the limit points (x1(∞),y1(∞)),…,(xn(∞),yn(∞)). Therefore the limit megapoint 𝒫(∞) belongs to the megaset 𝔐(𝕄). This proves that the latter is closed.
Megaset for Lines
Let 𝕄L consist of all lines in ℝ2. The corresponding megaset 𝔐(𝕄L)⊂ℝ2n is described in [28, Chapter 10]. A megapoint (x1,y1,…,xn,yn) belongs to 𝔐(𝕄L) if and only if all the n planar points (x1,y1),…,(xn,yn) belong to one line (i.e., they are collinear). This condition can be expressed by Cn,3 algebraic relations as follow:
(8.8)det[xi-xjyi-yjxi-xkyi-yk]=0
for all 1≤i<j<k≤n. Each of these relations means that the three points (xi,yi), (xj,yj), and (xk,yk) are collinear; cf. (6.1). All of these relations together mean that all the n points (x1,y1),…,(xn,yn) are collinear.
Note that 𝕄L is specified by n-2-independent relations; hence it is an (n+2)-dimensional manifold (algebraic variety) in ℝ2n. The relations (8.8) are quadratic, so 𝕄L is a quadratic surface. It is closed in topological sense; hence the problem of finding the best-fitting line always has a solution.
Megaset for Circles
Let 𝕄C consist of all circles in ℝ2. The corresponding megaset 𝔐(𝕄C)⊂ℝ2n is described in [4, Section 3.4], and we briefly repeat it here. A megapoint (x1,y1,…,xn,yn) belongs to 𝔐(𝕄C) if and only if all the n planar points (x1,y1),…,(xn,yn) belong to one circle (we say that they are “cocircular’’). In that case all these points satisfy one quadratic equation of a special type as follow:
(8.9)A(x2+y2)+Bx+Cy+D=0.
This condition can be expressed by Cn,4 algebraic relations as follow:
(8.10)det[xi-xjyi-yjxi2-xj2+yi2-yj2xi-xkyi-ykxi2-xk2+yi2-yk2xi-xmyi-ymxi2-xm2+yi2-ym2]=0
for 1≤i<j<k<m≤n. Note that (8.10) include megapoints satisfying (8.8); hence they actually describe the union 𝔐(𝕄C)∪𝔐(𝕄L).
The determinant in (8.10) is a polynomial of degree four, and 𝔐(𝕄C)∪𝔐(𝕄L) is an (n+3)-dimensional algebraic variety (manifold) in ℝ2n defined by quartic polynomial equations. Note that the dimension of 𝔐(𝕄C)∪𝔐(𝕄L) is one higher than that of 𝔐(𝕄L), that is,
(8.11)dim𝔐(𝕄C)∪𝔐(𝕄L)=dim𝔐(𝕄L)+1.
A closer examination shows that 𝔐(𝕄L) plays the role of the boundary of 𝔐(𝕄C), that is, 𝔐(𝕄C) terminates on 𝔐(𝕄L). The megaset 𝔐(𝕄C) is not closed, but if we add its boundary 𝔐(𝕄L) to it, it will become closed.
Megaset for Ellipses
Let 𝕄E consist of all ellipses in ℝ2. The corresponding megaset 𝔐(𝕄E)⊂ℝ2n can be described in a similar manner as above. A point (x1,y1,…,xn,yn) belongs in 𝔐(𝕄E) if and only if all the n planar points (x1,y1),…,(xn,yn) belong to one ellipse (we say that they are “coelliptical’’). In that case all these points satisfy one quadratic equation of a general type as follow:
(8.12)Ax2+2Bxy+Cy2+2Dx+2Ey+F=0.
This equation actually means that the points belong to one conic (either regular or degenerate). This condition can be expressed by Cn,6 algebraic relations as follow:
(8.13)det[xi-xjyi-yjxi2-xj2yi2-yj2xiyi-xjyjxi-xkyi-ykxi2-xk2yi2-yk2xiyi-xkykxi-xmyi-ymxi2-xm2yi2-ym2xiyi-xmymxi-xlyi-ylxi2-xl2yi2-yl2xiyi-xlylxi-xryi-yrxi2-xr2yi2-yr2xiyi-xryr]=0
for 1≤i<j<k<m<l<r≤n. Each of these relations means that the six points (xi,yi), (xj,yj), (xk,yk), (xm,ym), (xl,yl), and (xr,yr) satisfy one quadratic equation of type (8.12); that is, they belong to one conic (either regular or degenerate). All of these relations together mean that all the n points (x1,y1),…,(xn,yn) satisfy one quadratic equation of type (8.12); that is, they all belong to one conic (regular or degenerate). Therefore the relations (8.13) describe a much larger megaset 𝔐(𝕄Conics) corresponding to the collection of all conics, regular and degenerate, that is,
(8.14)𝕄Conics=𝕄E∪𝕄H∪𝕄∪∪𝕄LL
using our previous notation.
The determinant in (8.13) is a polynomial of the eighth degree, and 𝔐(𝕄Conics) is a closed (n+5)-dimensional algebraic manifold in ℝ2n. It is mostly made of two big megasets: 𝔐(𝕄E) and 𝔐(𝕄H) and they both are (n+5)-dimensional. Other megasets (parabolas and pairs of lines) have smaller dimensions and play the role of the boundaries of the bigger megasets 𝔐(𝕄E) and 𝔐(𝕄H).
Illustrations
The structure of the megaset 𝔐(𝕄Conics) is schematically depicted in Figure 7, where it is shown as the xy plane {z=0} in the 3D space (the latter plays the role of the megaspace ℝ2n). The positive half-plane H+={y>0,z=0} represents the elliptic megaset 𝔐(𝕄E), and the negative half-plane H-={y<0,z=0} represents the hyperbolic megaset 𝔐(𝕄H). The x-axis {y=z=0} separating these two half-planes represents the lower-dimensional megasets 𝔐(𝕄∪∪𝕄LL) in the decomposition (8.14). True, the real structure of 𝔐(𝕄Conics) is much more complicated, but our simplified picture still shows its most basic features.
Schematic illustration of the megaset 𝔐(𝕄Conics).
Now recall that finding the best-fitting conic corresponds to an orthogonal projection of the given megapoint 𝒫 in the megaspace ℝ2n (in our illustration, it would be a point (x,y,z)∈ℝ3) onto the megaset 𝔐(𝕄Conics) (in our illustration—onto the xy plane). In Figure 7 the point (x,y,z) is simply projected onto (x,y,0). What are the chances that the footpoint of the projection corresponds to a “boundary’’ object in 𝕄∪∪𝕄LL (i.e., to a secondary object)? Clearly, only the points of the xz plane {y=0} are projected onto the line {y=z=0}. If the point (x,y,z)∈ℝ3 is selected randomly with an absolutely continuous distribution, then a point on the xz plane would be chosen with probability zero. This fact illustrates the sufficiency of the model collection of nondegenerate conics (even the sufficiency of ellipses and hyperbolas alone, without parabolas); the best-fitting object will be a primary object with probability one.
But what if our model collection consists of ellipses only, without hyperbolas? Then in our illustration, the corresponding megaset would be the positive half-plane H+={y>0,z=0}. Finding the best-fitting ellipse would correspond to an orthogonal projection of the given point (x,y,z)∈ℝ3 onto the positive half-plane H+={y>0,z=0}. Now if the given point (x,y,z) has a positive y-coordinate, then it is projected onto (x,y,0), as before, and we get the desired best-fitting ellipse. But if it has a negative y-coordinate, then it is projected onto (x,0,0), which is on the boundary of the half-plane, so we get a boundary footpoint; that is, a secondary object will be the best fit. We see that all the points (x,y,z) with y<0 (making a whole half-space!) are projected onto the boundary line, hence for all those points the best-fitting ellipse would not exist! This fact clearly illustrates the deficiency of the model collection of ellipses.
One may wonder: How is it possible that the collection of circles is sufficient (as we proved in Section 5), while the larger collection of ellipses is not? Indeed every circle is an ellipse, hence the collection of ellipses contains all the circles. So why the sufficiency of circles does not guarantee the sufficiency of the bigger, inclusive collection of ellipses? Well, this seemingly counterintuitive fact can be illustrated too (see Figure 8).
Schematic illustration of the megaset 𝔐(𝕄C)⊂𝔐(𝕄E).
Suppose that the megaset 𝔐(𝕄C) for the collection of circles is represented by the set U={y=x2,x≠0,z=0} in our illustration. Note that U consists of two curves (branches of a parabola on the xy plane), both lie in the half-plane H+={y>0,z=0} that corresponds to the collection of ellipses. So the required inclusion U⊂H+ does take place. The two curves making U terminate at the point (0,0,0), which does not belong to U, so it plays the role of the boundary of U.
Now suppose a randomly selected point (x,y,z)∈ℝ3 is to be projected onto the set U. What are the chances that its projection will end up on the boundary of U, that is, at the origin (0,0,0)? It is not hard to see (and prove by elementary geometry) that only points on the yz plane may be projected onto the origin (0,0,0) (and not even all of them; points with large positive y-coordinates would be projected onto some interior points of U). So the chance that the footpoint of the projection ends up at the boundary of U is zero. This illustrates the sufficiency of the smaller model collection of circles, despite the deficiency of the larger model collection of ellipses (which we have seen in Figure 7).
Appendix
Here we prove the theorem on no local minima for n=5 (Section 6).
Proof.
Suppose that S is a conic on which ℱ takes a local minimum. We will assume that S is nondegenerate; that is, S is an ellipse, or hyperbola, or parabola. Degenerate conics are easier to handle, so we omit them.
Let P1,P2,P3,P4,P5 denote the given points and Q1, Q2, Q3, Q4, and Q5 their projections onto S. Since S does not interpolate all the given points simultaneously, there exists at least one which is not on S. Suppose that it is P1, hence P1≠Q1.
Let us first assume that Q1 is different from the other projections Q2,Q3,Q4,andQ5. Then let Q1′ be a point near Q1 such that dist(Q1′,P1)<dist(Q1,P1). In other words, we perturb Q1 slightly by moving it in the direction of P1. Then there exists another non-degenerate conic S' that interpolates the points Q1′,Q2,Q3,Q4,andQ5. It is easy to see that
(A.1)dist(P1,S′)≤dist(P1,Q1′)<dist(P1,Q1)=dist(P1,S),dist(Pi,S′)≤dist(Pi,Qi)=dist(Pi,S)foreveryi=2,…,5.
As a result,
(A.2)ℱ(S′)=∑i=15[dist(Pi,S′)]2<∑i=15[dist(Pi,S)]2=ℱ(S),
as desired. Since Q1′ can be placed arbitrarily close to Q1, the new conic S' can be made arbitrarily close to S.
Now suppose that the projection point Q1 coincides with some other projection point(s). In that case there are at most four distinct projection points. We will not move the point Q1, but we will rotate (slightly) the tangent line to the conic S at the point Q1. Denote that tangent line by T. Note that T is perpendicular to the line passing through P1 and Q1.
Now let T′ be another line passing through the point Q1 and making an arbitrarily small angle with T. It is not hard to show, by elementary geometry, that there exists another nondegenerate conic S′ passing though the same (≤4) projection points and having the tangent line T′ at Q1.
Since the tangent line T′ to the conic S′ at the point Q1 is not orthogonal to the line P1Q1, we easily have
(A.3)dist(P1,S′)<dist(P1,Q1)=dist(P1,S).
At the same time we have
(A.4)dist(Pi,S′)≤dist(Pi,Qi)=dist(Pi,S)foreveryi=2,…,5.
As a result, consider
(A.5)ℱ(S′)=∑i=15[dist(Pi,S′)]2<∑i=15[dist(Pi,S)]2=ℱ(S),
as desired. Since the line T' can be selected arbitrarily close to T, the new conic S' will be arbitrarily close to S.
Acknowledgment
N. Chernov was partially supported by National Science Foundation, Grant DMS-0969187.
AhnS. J.Geometric Product Specification (GPS)Acceptance and representation test for coordinate measuring machines (CMM)—Part 6: estimation of errors in computing Gaussian associated featuresChengC.-L.Van NessJ. W.ChernovN.AdcockR. J.Note on the method of least squaresKummellC. H.Reduction of observation equations which contain more than one observed quantityMadanskyA.The fitting of straight lines when both variables are subject to errorVillegasC.Maximum likelihood estimation of a linear functional relationshipWaldA.The fitting of straight lines if both variables are subject to errorAndersonT. W.Estimation of linear functional relationships: approximate distributions and connections with simultaneous equations in econometricsAndersonT. W.SawaT.Exact and approximate distributions of the maximum likelihood estimator of a slope coefficientRobinsonS. M.Fitting spheres by the method of least squaresThomA.A statistical examination of the megalithic sites in BritainAlbanoA.Representation of digitized contours in terms of conic arcs and straight-line segmentsBiggerstaffR. H.Three variations in dental arch form estimated by a quadratic equationBooksteinF. L.Fitting conic sections to scattered dataPatonK.Conic sections in chromosome analysisTaubinG.Estimation of planar curves, surfaces, and nonplanar space curves defined by implicit equations with applications to edge and range image segmentationPeiS. C.HorngJ. H.Optimum approximation of digital planar curves using circular arcsSarkarB.SinghL. K.SarkarD.Approximation of digital curves with line segments and circular arcs using genetic algorithmsNievergeltY.A finite algorithm to fit geometrically all midrange lines, circles, planes, spheres, hyperplanes, and hyperspheresBermanM.CulpinD.The statistical behaviour of some least squares estimators of the centre and radius of a circleChernovN.LesortC.Least squares fitting of circlesZelnikerE. E.ClarksonI. V. L.A statistical analysis of the Delogne-Kåsa method for fitting circlesNievergeltY.Fitting conics of specific types to dataMateiB.MeerP.Reduction of bias in maximum likelihood ellipse fitting3Proceedings of the 15th International Conference on Computer Vision and Pattern Recognition (ICCVPR '00)September 2000Barcelona, Spain802806FitzgibbonA.PiluM.FisherR. B.Direct least square fitting of ellipsesMalinvaudE.