The positioning of the apple growth cycle plays a very important role in predicting the development of apples and guiding fruit farmers in agricultural operations. The traditional method of manually positioning the apple growth cycle has problems such as low efficiency and poor accuracy. Pattern recognition provides support for continuous and rapid positioning during the apple growth process. Under the natural conditions of the orchard, due to the large differences in the individual colors of the apples during the growth process and the influence of factors such as light changes, the photographed apple images are more complex, which brings certain difficulties to the segmentation and recognition of the apples. In this paper, pattern recognition is used to automatically identify and extract the growth stages of apples, a hue intensity (HI) color segmentation algorithm based on a Gaussian distribution model based on prior knowledge is studied, and then an active shape model (ASM) is used to identify each period of apple growth based on pattern recognition. After a series of experimental verifications, the ASM-based automatic identification method proposed in this paper is feasible and can identify the various growth periods of apples, thereby serving the mechanized production of apples.
Apples have always been recognized as a fruit that can prevent a variety of diseases. Doctors said, “A study by the University of California in the United States has found that apples contain a flavonoid substance that can delay cell aging, protect the heart, and prevent cerebrovascular diseases.” In 2020, China has become the world’s largest apple producer, with apple planting area and output accounting for more than 50% of the world. The apple industry has brought good economic benefits and has become the main source of income for farmers. Precisely positioning the growth cycle of apples to grasp its development status is of great importance for guiding production management such as fertilization and irrigation. In the growth process of apples, it is important to accurately assess the growth of apples and perform appropriate work at the appropriate time.
Smart agriculture is the integration between agriculture and Artificial Intelligence (AI) technology and modern information technology, and it is also a technology that is urgently needed in the development of global agriculture. Nondestructive monitoring of crop growth is one of the important links in smart agriculture. It requires real-time and accurate acquisition of plant growth information to guide the fine management of crops. In particular, the monitoring of crops in different growth periods is an important part of crop growth monitoring. First of all, understanding the growth period of crops can help people analyze the relationship between field crop growth and environmental conditions; in addition, it can effectively guide operations, thereby significantly improve the level of agricultural mechanization and crop yields [
Pattern recognition can detect subtle changes in plants due to malnutrition earlier than human vision, providing a reliable basis for timely irrigation and supplementation of nutrients. Under the current circumstances, if apple’s economic benefits and automated production are effectively improved, this will have important practical significance. In the process of mechanized production, orchard management is a very important task. The apple’s growth cycle is divided into six major periods, the first is the growth period of the seedlings; the seedlings need to be carefully maintained and provided with appropriate water and fertilizer management to make the apples grow better. Next is the flower bud differentiation period, and attention should be paid to temperature and light control. The temperature is 12∼18°C; during the flowering period from April to June, stop watering to avoid falling flowers; then in the flowering period of apples, supply enough water before flowering, and stop watering at the flowering period to prevent the flowers from falling off. In the case of poor growth, it is necessary to cut the flower buds in time; cut off the bad flowers and poorly grown flowers to maintain a healthy growth state. This is followed by the physiological fruit drop period and the fruit expansion period of the apple, and the maintenance work should be done well. Finally, it should be fertilized and pruned in time during its maturity period to allow it to grow and produce better results. In actual production, the external morphological structure of the crop has undergone tremendous changes from seedling growth to maturity, and these morphological and structural changes are also very significant for the image. Therefore, it is very necessary and urgent to carry out pattern recognition research on the growth and development cycle of apples with the help of computer vision and image processing theories and methods.
The main contributions of this paper are summarized as follows: (i) we use the HI color based on Gaussian distribution, which is suitable for the image segmentation algorithm of apples at each period of the growth process in the complex environment of the orchard to perform apple image segmentation. (ii) On the basis of correct segmentation, an ASM based on pattern recognition is used to identify apples in the growth process of each period. In other words, the novelty of this paper is mainly to use the HI color based on Gaussian distribution to perform apple image segmentation, and then use ASM to identify the whole apple growth cycle from seedling stage to mature fruit.
The rest of the paper is organized as follows: Section
Nowadays, labor on farms and orchards depends largely on skilled farmers. Manual labor wastes time and increases the cost of production, while uneducated and inexperienced workers make unnecessary mistakes. In [
Relying on the continuous development of image processing technology and pattern recognition technology, we can use computer technology for crop shape recognition. At present, in crop morphology recognition, research mainly focuses on the appearance characteristics of crops, that is, the measurement of the geometric shape characteristics of the crops, for example, the classification of certain crops by different leaf shape characteristics, including the rectangularity, circularity, and circularity of the leaves. Geometric parameters such as eccentricity, leaf size and invariant moments, and crop classification and identification can also be achieved through crop flowers, fruits. Most of the work done is focused on the study of specific organ morphology of crops, such as leaves and flowers. Few studies involve the use of computer vision systems to automatically identify the growth and development stages of crops. However, the use of pattern recognition to locate the growth cycle of apples is an indispensable part of practical applications, which provides impetus for the development of this paper. Especially different from the previous work, the novelty of this paper is mainly to use the HI color based on Gaussian distribution to perform apple image segmentation, and then use ASM to identify the whole apple growth cycle from seedling stage to mature fruit.
Computer vision usually includes the following modules: light source, camera or camera image capture card, input and output module, and image processing software. The performance of the image acquisition system will directly affect the quality of the collected data and then affect the subsequent image processing and analysis. Taking into account the actual situation of this research and the goals that need to be completed, the hardware system for apple image segmentation and for identification of each key developmental stage is mainly composed of a computer and an image acquisition system. The image acquisition is usually done by a video camera, ordinary digital or optical camera, and then the digital image in the memory card is converted to the computer. This paper will mainly use digital cameras for data acquisition. The image acquisition system mainly includes tripods, digital cameras, computers, and data transmission equipment. This paper can obtain apple images of different weather conditions and different growth stages through this system and then identify them through image processing related algorithms.
The subject of this paper is apples, and the selected variety is Fuji apple. The image collection time is mainly determined according to the growth status and growth rules of the apple. As far as this research is concerned, it is necessary to consider accurately segmenting apples from images under different weather conditions and automatically identify the four key growth stages of apple flower bud differentiation, apple flowers, physiological fruit drop, and fruit expansion in the orchard. Therefore, we need to shoot continuously at specific times every day. The shooting object is apple (the growth cycle starts from the seedling to the end of the mature period). Apple image collection is mainly concentrated in two time periods. The first time period is mainly to photograph the apples in the orchard. Its function is to analyze the relationship between the green information of the apple and the light under different weather conditions and to lay the foundation for segmentation; the latter time period is mainly to obtain a single apple, and its role is to identify later created training samples. The shooting environment changes include dark light to strong light, weather conditions (sunny, rainy, and cloudy), shooting time (morning, noon, and afternoon), and dry to humid soil. It is also necessary to adjust the height and angle of the camera in time to find a reasonable configuration that can obtain high-quality image data.
The accurate segmentation and extraction of apples in the image constitute a prerequisite for realizing apple growth monitoring. However, the completion of this work is affected by many factors, such as changes in light conditions, changes in surface soil, and complex environments. In order to achieve accurate segmentation of apple images in complex environments, this paper studies an HI color segmentation algorithm based on Gaussian distribution model based on prior knowledge of apples in different weather conditions and different growth and development stages. The algorithm only needs a small number of training samples to complete the segmentation of the apple image.
In the algorithm of crop image segmentation, the role of color index is very important. The general color index algorithms mainly include ExG (Excess Green—over-green); ExR (Excess Red—over-red); ExG–ExR (Excess Green minus Excess Red—over-green minus over-red); CIVE (Color Index of Vegetation Extraction—a vegetation color index extraction); and VEG (Vegetation). These algorithms only need to use the three components of red, green, and blue in the RGB color space. The calculation is simple, and it is not sensitive to changes in light intensity; even to complex environments, change has a certain degree of adaptability. However, the prerequisite for the realization of the algorithm is to normalize the color space and finally achieve crop extraction by setting the segmentation threshold.
The method of normalizing the color space is shown in (
On the basis of
The calculation of the color index is based on the calculation of
The HSI color model expresses the way the human visual system perceives the color of an object, where hue (H) represents the difference in color, saturation (S) represents the purity of the color, and intensity (I) represents the brightness of the color. These characteristic components are used to describe the color. The use of the HSI color model in crop segmentation has the following advantages: compared with the RGB color model, its three color feature components can be directly used for segmentation separately; at the same time, it overcomes the influence of lighting conditions [
The color distribution of a single-color object changes with intensity on the hue-saturation plane, where a single color is different in each period of apple growth, and it is green in the growth period of the seedling and the physiological fruit drop period; in the flower bud, it is pink at the differentiation stage, yellow at the flowering stage of the apple, and red at the fruit expansion and maturity stage. Therefore, when building a single-color pixel model in the HSI color model, it is necessary to adapt to changes in external lighting conditions. Therefore, a single Gaussian distribution model can be used to represent the distribution of tone values at a certain intensity. The probability density function of Gaussian distribution is shown in
Here,
Here,
The HI segmentation algorithm used in this paper is based on the HSI color model, and the type of the apple image collected in the experiment is the RGB color model. Therefore, the image needs to be converted from the RGB color model to the HSI color model when building the tone intensity table and before image segmentation. The basic principle of the conversion from RGB color model to HSI color model is as follows: First, the brightness factor needs to be separated from the RGB color model; the chroma is divided into two parts, hue and saturation; and the angle vector is expressed as hue. On the whole, it is a conversion from a unit cube based on Cartesian coordinates to a double cone based on cylindrical polar coordinates [
The values of
Set a pixel
Here,
The identification of each period in the apple growth process is carried out on the basis of accurately segmenting the apples. After image processing, the influence of other factors on the identification effect is reduced. Researchers at home and abroad mainly use information such as shape parameters, color, and texture characteristics to identify single leaves of plants; to identify flower bud differentiation, flowering period, and fruit period, they count on the basis of identifying single leaves and then determine which period the plant is in. In this study, ASM method was used to identify the growth cycle of apples, taking into account the overall geometric shape of the apples in each period and reducing the influence of different leaves on the identification results.
ASM is a statistical model method applied to complex shape matching proposed by scholars [
We use
When manually calibrating the target contour feature points, it is necessary to ensure that the calibration sequence of the feature points in each image is consistent with the calibration position. Otherwise, the alignment of the training samples in the next step will be affected, and the modeling will be unreasonable. There are different shooting angles and different shooting distances during the data acquisition of a certain key growth period of apples. Therefore, the positions and scales of the apples in the images are different, which causes the distribution of training samples in the two-dimensional space to be disordered, resulting in failures. This shows the changing law of the shape of an apple during a key growth period, so it is unreasonable to directly count and process the training samples. Therefore, it is necessary to perform standardized alignment processing on the manually labeled training samples. The main operation steps are as follows: First select a certain sample shape as the basic shape, and other samples are as close as possible to the basic shape through the steps of scale enlargement and reduction, rotation, and translation.
If we only align two samples, their corresponding shape vectors are
By the least square method, one obtains
Partially differentiate
Therefore, it can be extended to the alignment of
The average model in the shape model is defined as follows:
The final shape model in the training set can be expressed by the sum of the changes of the average shape model and the weight model.
Here,
The change of the shape weight
Here,
The establishment of the local gray-scale texture model firstly needs to analyze the gray-scale information [
At the same time, in order to reduce the influence of different lighting conditions, we have done a normalization process on the basis of the gradient value
Then, calculate the average gray information model
Calculate the covariance matrix of the feature point
According to the calculation of the above formula, we can get the local gray information characteristics of each model feature point: the average gray information model
Mahalanobis distance
When searching for a target in an image, the first step is to use artificially marked feature points to obtain the initial shape model, then use the local gray-scale texture model to select the best matching point, and combine the limitations and changes of shape parameters to obtain a matching result map. The specific process is as follows: The model is initialized and positioned. In the target image, whether there are apple seedlings, flower buds, and fruits at various stages of the growth process, generally determine the approximate position first; on this basis, match the seedling, flower bud, and fruit stage shapes of the initial shape model to obtain the approximate position of the target image; the initial shape is Discover new model shapes. Search in the vicinity of the feature point according to the normal direction of the feature, and calculate the minimum Mahalanobis distance The shape parameters are updated. The displacement required by the new model to the characteristic target point is obtained by affine transformation. Shape parameter constraints. From the probability distribution model, the variation range of the shape parameter is Repeat the steps from (
When the ASM searches for the best matching point, it is often affected by internal and external factors, such as the initial target location and the noise of the image itself. In turn, deviations occur when each feature point is matched with the best candidate point corresponding to it, and the search performance is changed. The result of the search becomes worse as the number of deviation points increases or the deviation increases, which in turn affects the matching effect of the ASM algorithm. The traditional ASM algorithm faces the following contradictions: In the process of feature point matching, from one perspective, the expected search range is large enough to ensure that the best matching position of the feature point can be searched. However, if the search interval is too large, the number of nonoptimal feature points similar to ideal feature matching points in the search interval will increase, the computational efficiency of the algorithm will be greatly reduced, and the possibility of errors will continue to increase. From another perspective, in general, the smaller search interval is expected to reduce the amount of calculation and finally improve the efficiency. In response to the abovementioned contradictions, Cootes and other research scholars proposed a new ASM search method, namely, multiresolution search strategy. The basic idea is “from low to high” multilevel search: The first step is to search for the approximate position of the target in the lowest resolution image. Then, in the image environment with the resolution gradually increasing, the precise position of the target is continuously improved until the original image that is the highest resolution image is found. This method can speed up the running speed and reduce the probability that the local structure of the image interferes with the shape model.
In the process of multiresolution search strategy, the first step is to build a pyramid of Gaussian images. The images in each layer are the same, but the difference between them is different resolutions. The lowest layer which is the original image is image layer 0, and the upper layer of lower resolution image is image layer 1. The method of acquisition is to process the lowest layer image with a 5
In order to determine the appropriate threshold in the HI color segmentation of the Gaussian distribution model to accurately segment the apples in the growth process of each period, this paper selected 45 different weather conditions (sunny, cloudy, and rainy), different environments, and different growth stages. The apple image of the period was used as a sample for testing. At the same time, the relationship between the selection of
Each period of apple growth.
HI segmentation result of Gaussian distribution model when
HI segmentation result of Gaussian distribution model when
HI segmentation result of Gaussian distribution model when
In order to further prove the excellent performance of the HI color segmentation algorithm of the Gaussian distribution model in extracting apples in the growth process of each period, this paper again selects typical data from the obtained apple image data: different weather conditions (sunny, cloudy, and rainy), different growth stages, and different levels of complexity are used as test samples.
In this study, the experimental results of the four algorithms of CIVE, ExG–ExR, ExG, and VEG are compared with the experimental results of the method in this paper. The threshold of the HI segmentation algorithm of the Gaussian distribution model is set to
The ASM algorithm is used to accurately describe the overall two-dimensional shape of the apple during the growth process, and the multiresolution search strategy is used to determine the position of the apple during the growth process to identify it. The training samples are shown in Figure
Outline of markings at each period of apple growth.
In order to improve the accuracy and efficiency of recognition at each period of the apple growth process, we should pay attention to the following issues. The outline needs to be marked in order, and the calibration points should be distributed as far as possible on the edge of the contour. Use these ordered calibration points to describe the contour, and then you can convert them from the image to the two-dimensional space and record it to describe the overall contour.
Although the contour points for calibration at each period of apple growth process are marked in order, there are still certain errors. The description of the overall geometric shape will still be affected by three aspects: When obtaining the contour calibration points at each period of apple growth process, the original calibration points are in their respective coordinate systems. Therefore, the apples of the same stage and different periods have different coordinate systems, and there will be differences when they are unified to the same coordinate system. When the data is collected, the distance between the camera and the apple is different, which causes the scales of apples in different stages and different periods to be inconsistent in the image, so they cannot be directly compared. The apple and the imaging plane are not completely parallel, which causes distortion in the image of the geometry of the apple at each stage of the growth process. In order to overcome the influence of the abovementioned unfavorable factors, it is necessary to align the sequence points of the apple contour calibration at each stage in different images. The main purpose of alignment is to be able to compare the same marked feature points of training samples from different coordinate systems and build a statistical shape model on this basis to objectively describe the trend and law of shape changes.
After the calibration points of the training samples are aligned, the average sequence calibration points of the training samples are obtained according to (
In this paper, we select 100 apple trees in the orchard and segment the HI image based on the Gaussian distribution model. The optimal
The success rate of apple identification and positioning at each period of the growth process is shown in Figure
The positioning success rate of each period of apple growth.
Based on pattern recognition, this paper studies an HI color segmentation algorithm based on Gaussian distribution, which is suitable for apple image segmentation at each period of the growth process in the complex environment of the orchard. It is based on prior knowledge to segment the apple image. First, the apple image with the background removed under different weather conditions is obtained as the training sample, the HI look-up table is established, and then each pixel in the image is judged on whether it belongs to an apple. Morphological processing and contour tracking are performed on the binary image of the segmented apple image, and the appropriate threshold is determined in the HI color segmentation of the Gaussian distribution model to accurately segment the apples in the growth process of each period. On the basis of correct segmentation, the method based on pattern recognition, ASM, is used to identify the apples in the growth process of each period; the overall outline is calibrated to describe their shape; and then the training samples are counted and calculated. Finally, the initial shape model is obtained. This paper shows, through a series of experimental results, that our algorithms are effective and feasible; they can meet actual production needs, provide decision support for real-time management of orchards, and finally lay the foundation for the application of smart agriculture. Although this paper has achieved some satisfactory results, it also has shortcomings. Using the Gaussian distribution HI color segmentation algorithm to segment the apple image depends on the selection of the
The data used to support the findings of this study are available from the corresponding author upon request.
The authors declare that they have no conflicts of interest.
This work was supported by the National Natural Science Foundation of China (no. 31860331), YEFICRC Project of Yunnan Provincial Key Programs (no. 2019ZG009), and Yunnan Agricultural Joint Youth Fund (no. 2018FG001-091).