Online Measuring and Size Sorting for Perillae Based on Machine Vision

. Perillae has attracted an increasing interest of study due to its wide usage for medicine and food. Estimating quality and maturity of a perillae requires the information with respect to its size. At present, measuring and sorting the size of perillae mainly depend on manual work, which is limited by low e ﬃ ciency and unsatis ﬁ ed accuracy. To address this issue, in this study, we develop an approach based on the machine vision (MV) technique for online measuring and size sorting. The geometrical model and the corresponding mathematical model are built for perillae and imaging, respectively. Based on the built models, the measuring and size sorting method is proposed, including image binarization, key point determination, information matching, and parameter estimation. Experimental results demonstrate that the average time consumption for a captured image, the average measuring error, the variance of measuring error, and the overall sorting accuracy are 204.175ms, 1.48 mm, 0.07mm, and 93%, respectively, implying the feasibility and satis ﬁ ed accuracy of the proposed approach.


Introduction
Perillae is widely distributed throughout Asia, and it has attracted an increasing interest in the field of medicine and pharmacy.Perillae has been applied for bacteriostatic, detoxifying, antitussive, and phlegm for thousands of years in traditional Chinese medicine.Recently, more attentions have been paid to the study with respect to perillae, including the perillae herba ethanolic [1], the luteolin [2], and the rosmarinic acid [3].Besides, perillae is also a popular food in countries such as China, Japan, and Korea.The size of perillae indicates its quality and maturity that closely relates to its medical and edible value.Hence, it is of great significance to measure and sort the size of perillae in advance.At present, measuring and sorting the size of perillae mainly depend on manual work, which is of low efficiency and unsatisfied accuracy.Therefore, it is valuable to develop an approach for accurate online measurement and sort of perillae.
Machine vision (MV) is the technique to provide imaging-based automatic inspection [4][5][6], and it has been widely used in the fields of industry, such as object detection and robot guidance [7,8].For the applications in agriculture, previous work has explored into various crops [9,10].In [11], a compact machine vision system based on hyperspectral imaging and machine learning is presented to detect aflatoxin in chili pepper.In [12], a hierarchical grading method is applied for real-time defect detection and size sorting of potatoes.In [13], a study is conducted to predict ripening quality in mangoes using RGB images, for which the hierarchical clustering method is employed to classify the ripening period into five stages based on quality parameters.In [14], the authors combine the MV and the support vector machine to develop an intelligent system for sorting of peeled pistachio kernels and shells.In [15], Sabliov et al. develop an MV-based method to measure volume and surface area of ellipsoidal agricultural products by regarding the objects as the sum of superimposed elementary frustums of right circular cones.In [16], Yao et al. develop real-time detection instrumentation for aflatoxin-contaminated corn using a narrow-band fluorescence index.In [17], Pedreschi et al. present an inexpensive computer vision system for measuring the color of a highly heterogeneous food material such as potato chips.In [18], Huang et al. propose an approach for identification of defect pleurotus geesteranus based on computer vision.In [19], Sun et al. develop a machine vision system and the dynamic weighing system for the measurement of egg external physical characteristic and weight, such that the nondestructive method for online estimation of egg freshness is achieved.Additionally, a set of studies have been conducted on measurement, cabbage [20], flower mushroom [21], cherry [22], litchi [23], and other agricultural products [24][25][26][27][28][29].
However, the automatic online detection of perillae has not been explored till now.The geometry model and the corresponding mathematical model of perillae used for automatic detection have not been built yet.The size sorting of perillae still requires manual work.To address this issue, in this study, we develop a novel approach for online measuring and size sorting of perillaes.A charge-coupled device (CCD) camera is employed to acquire the images of perillaes in the lighting box, and the MV method is proposed for image processing.The contributions of this study can be summarized as follows.
(1) We build the general geometry model of perillae and the corresponding mathematical model for image processing (2) We propose the MV-based method for accurate online measuring and size sorting of perillae under the proposed models (3) We develop the practical system for our theoretical method, the feasibility and accuracy of which are verified by the experimental results The remainder of this paper is organized as follows.Section 2 introduces the proposed geometry model and mathematical model.Section 3 provides the MV-based online measuring method.Section 4 presents the experimental results and analysis.Section 5 concludes this paper.

Geometry and Mathematical Models
2.1.Geometry Model. Figure 1 shows a typical perillae for medical and food usage.The notation L u is used to denote the length of a perillae, i.e., the maximum size of perillae leaf.We mainly consider the length between P A and P B as it is the useful part of perillae, regardless of the medical or food usage.The length between P B and P C , i.e., the length of perillae, is not taken into the measurement.
Figure 2 presents the sorting principle for the size of perillae.In this study, the sizes are divided into four grades according to L u .As shown in Figure 2, the first grade is expressed as "Spec M" with the size of 7 cm ≤ L u < 8:5 cm, denoting the smallest perillae.The second grade, "Spec L(s)," and the third grade, "Spec L(b)," are determined by 8:5 cm ≤ L u < 9:5 cm and 9:5 cm ≤ L u < 10:5 cm, respectively.The fourth grade, expressed as "Spec 2M," is used to represent the largest size, given by 10:5 cm ≤ L u < 12 cm.

Mathematical Model.
Next, we provide the mathematical model for imaging of perillae.The camera used in this study was calibrated before it is employed for imaging.As shown in Figure 3, we captured 20 chessboard images with different poses and then utilized the calibration tool in the OpenCV to calibrate the camera.
Let u, v, and w denote the horizontal coordinate, the vertical coordinate, and the position perpendicular to the chessboard in the camera coordinate, respectively, and let x, y, and z denote the corresponding coordinates in the world coordinate.The calibration results demonstrate the distortion error, and the reprojection error can be ignored.Assume Mðx, y, zÞ is a pixel in the world coordinate system, and this pixel is then projected into the camera coordinate system, expressed  The matrix G is obtained through the calibration, and the result is By combining the geometrical model and (2), we can rewrite the size of perillae as follows: where p A,x , p A,y , p B,x , and p B,y are the coordinates of points P A and P B in the world coordinate system and q A,x , q A,y , q B,x , and q B,y are the coordinates of points P A and P B in the image coordinate system.Obviously, the required size can be calculated out directly if parameter z and the coordinates of P A and P B are known.Hence, we introduce the proposed method to determine this information in the next section.

Method for Measuring Perillae
In this section, we provide the proposed method for measuring and size sorting of perillae based on the MV and the proposed models.The method is composed of the following steps, binarizing the original image, determining the positions of P A and P B , matching information, and estimating parameter z.
where jðx, yÞ is the value of the pixel ðx, yÞ in the binarized image.The depth of the image is 8 bits, and therefore, the value of white pixels is set to 255.As indicated in (4), a pixel is determined to be white as long as 380 ≤ y ≤ 690, h b < 180, and h r < 220, where h b and h r are the values of the pixel in the blue channel and the red channel, respectively.The first condition, 380 ≤ y ≤ 690, ensures that the background is set to be black, and only the area containing perillaes is considered.The second and the third conditions are utilized to determine whether a pixel belongs to the area of perillae.The binarized image is shown as Figure 4(b).

Search for P A .
We first derive the contour from the binarized image.The strategy is to set the thresholds for the largest and the smallest areas of the perillaes.Thus, the areas that satisfy the constraint of thresholds are selected, and their contours are regarded as the contours of perillaes.This process can be expressed as where N denotes the total number of contours in the binarized image, A i denotes the area of the i th contours, and cjr i is the judging result.Then, the minimum enclosing circle and the convex hull of each perillae can be obtained.These perillaes are labelled as 1, 2, 3, and 4 from left to right based on the following judgement: where pjr j is the judgement for each position of perillae, c j,x denotes the horizontal coordinate of the center of the j th enclosing circle, and "Num1" denotes the amount of perillae in the captured image.The results are displayed in Figure 5(b) and are saved as "Pos1."Finally, we utilize the following strategy to find the endpoints of the l th contours, written as p 11,l and p 12,l : where cp l,k denotes the k th pixel in the l th convex hull, cp l,k,x and cp l,k,y are the coordinates of cp l,k , cc l,x and cc l,y are the coordinates of the center of the l th enclosing circle, r l represents the radius of the l th enclosing circle, and m l is the amount of pixels of the l th convex hull.In fact, p 11,l can be regarded as the point furthest away from the center of the enclosing circle, and p 12,l is the point furthest away from p 11,l .These two points are generally P A and P C .The data of "Num1," "Pos1," p 11 , and p 12 are saved as "Info1."

Search for P B .
To acquire the position of P B , we conduct one close operation, four erode operations, and four dilate operations in the OpenCV for the binarized image, successively.The results are presented in Figure 6(a).Then, we derive the treated contours using the strategy given by ( 7) and (8).The results are shown in Figure 6(b) and are saved as "Pos2."Finally, the minimum enclosing circle and the convex hull of the derived contours are obtained, shown as Figure 6(c).Similar to the procedure to search for P A and P C , two endpoints, p 21 and p 22 , are selected as the alternative  , respectively.Then, we compare the four distances and find out the two points that lead to the largest distance.Assuming the two points are p 12 and p 22 , we can draw the conclusion that P C is p 12 , as p 12 is the alternative point of P A and P C , and the distance from P C to P D is always larger than that of P A to P B .Repeat the procedure for all "Num1" contours of perillae, such that the position of P A and P B of each perillae can be obtained.We summarize all four possibilities in Table 1.

Scale Factor Estimation and Size Computation.
As indicated in (8), parameter z is the factor controlling the scale of imaging.Hence, we utilize the following scale method to estimate z.As shown in Figure 7, the original plane is z = c 2 .We use A 2 B 2 to represent the manual measurement and use A 0 B 0 to represent the corresponding measurement in the camera coordinate system.Then, we manually set z to c 1 to acquire the measurement A 1 B 1 by using (8).Based on these measurements, we have Therefore, z can be calculated by To eliminate the measurement error, we repeat the above process for 20 times, and the average value is used for the experiments, computed by where c 2i denotes the calculated value in the i th trial.The proposed approach is summarized in Figure 8.

Experimental Results
Our experimental system is shown in Figure 9, composed of a lighting box and a personal computer (PC).The lighting box contains four light-emitting diode (LED) light located as a ring on the top of the box and a CCD camera in the center of the box.The experimental setup is described as follows.
Before conducting the experiments, we manually set c 1 = 50 cm and obtain z = 68:765 cm by using the strategy introduced in Subsection 3.5.This value of z is used for all experiments in this study.First, we randomly chose 100 perillaes and manually measured their sizes.Then, we used the developed system to measure these perillaes.
The results by both manual measurement and automatic online measurement are presented in Figure 10.We computed the measurement error for each perillae, and the results are provided in Figure 11.Obviously, the results obtained by online measurement are close to those obtained by manual measurement.As provided in Table 2, among the 100 perillaes, the maximum measuring error (MAME) is 3.66 mm, while the minimum measuring error (MIME) is 0.02 mm.Most of the errors are lower than 3 mm.The overall average measuring error (OAME) and the variance of measuring error (VME) are 1.47 mm and 0.07 mm, respectively.

Conclusions
We developed an MV-based approach for automatic measuring and size sorting of perillae.We first built the geometric model for perillae and the mathematical model for imaging.Based on the models, the measuring and size sorting method was proposed, including image binarization, key point determination, information matching, and parameter estimation.We employed the CCD camera for imaging and the OpenCV tools for image processing.Experimental results have verified the feasibility of our system and its high accuracy of measuring and size sorting.By using 100 perillaes for experiments, the MAME and the MIME are 3.66 mm and 0.02 mm, respectively.Most of the errors are lower than 3 mm.The OAME  5 Journal of Sensors and the VME are 1.47 mm and 0.07 mm, respectively.Further study could address on the following issues.First, miniaturization for the developed system requires future work to make the proposed approach more particle.Second, utilizing the RGB image or hyperspectral image to detect the maturity of perillaes is also valuable for further study.

Figure 1 :Figure 2 :
Figure 1: Geometry model of perillae where P A , P B , and P C represent the apex of perillae, the separation point of leaf and petiole, and the endpoint of petiole, respectively.
close, erode, and dilate operations to binarized image Obtain the minimum enclosing circles and convex hullls of original contours Obtain the minimum enclosing circles and convex hullls of treated contours Find the alternative points of P A Find the alternative points of P B Calculate the Euclidean distances from alternative points Determine P A and P B

4
Journal of Sensors points of P B .The other one is defined as P D .The data of "Pos2," p 21 , and p 22 are saved into "Info2."3.4.Information Matching.Next, we should recognize P A and P B from p 11 , p 12 , p 21 , and p 22 .For this purpose, we first compute the Euclidean distance from p 21 to p 11 , p 21 to p 12 , p 22 to p 11 , and p 22 to p 12

Table 1 :
Information matching, P A and P B to p 11 , p 12 , p 21 , and p 22 .

Table 2 :
Overall measuring and sorting results.