Citrus fruits maturity detection in natural environments based on convolutional neural networks and visual saliency map

Citrus fruits do not ripen at the same time in natural environments and exhibit different maturity stages on trees, hence it is necessary to realize selective harvesting of citrus picking robots. The visual attention mechanism reveals a physiological phenomenon that human eyes usually focus on a region that is salient from its surround. The degree to which a region contrasts with its surround is called visual saliency. This study proposes a novel citrus fruit maturity method combining visual saliency and convolutional neural networks to identify three maturity levels of citrus fruits. The proposed method is divided into two stages: the detection of citrus fruits on trees and the detection of fruit maturity. In stage one, the object detection network YOLOv5 was used to identify the citrus fruits in the image. In stage two, a visual saliency detection algorithm was improved and generated saliency maps of the fruits; The information of RGB images and the saliency maps were combined to determine the fruit maturity class using 4-channel ResNet34 network. The comparison experiments were conducted around the proposed method and the common RGB-based machine learning and deep learning methods. The experimental results show that the proposed method yields an accuracy of 95.07%, which is higher than the best RGB-based CNN model, VGG16, and the best machine learning model, KNN, about 3.14% and 18.24%, respectively. The results prove the validity of the proposed fruit maturity detection method and that this work can provide technical support for intelligent visual detection of selective harvesting robots.