基于深度学习的葡萄果梗识别与最优采摘定位

针对葡萄采摘机器人在采摘作业中受果园环境干扰,难以准确识别与分割葡萄果梗及定位采摘点的问题,该研究根据葡萄生长的特点提出一种基于深度学习的葡萄果梗识别与最优采摘定位方法。首先通过改进掩膜区域卷积神经网络(Mask Region with Convolutional Neural Network,Mask R-CNN)模型对果梗进行识别与粗分割;然后结合阈值分割思想对果梗的色调、饱和度、亮度(Hue Saturation Value,HSV)色彩空间进行分段式提取,取每段色彩平均值作为该段果梗基准颜色阈值,利用区域生长算法对果梗进行精细化分割;最后计算果梗图像区域的质心,并以临质心点最近的果梗水平两侧中心作为最终采摘点。试验结果表明,在不同天气光照下该方法对葡萄果梗的检测精确率平均值为88%;在果梗成功识别后最优采摘点定位准确率达99.43%,单幅图像的果梗采摘定位平均耗时为4.90s,对比改进前Mask R-CNN检测耗时减少了0.99 s,F1-得分提高了3.24%,检测效率明显提升,该研究为葡萄采摘机器人提供了一种采摘点定位方法。

Recognition and the optimal picking point location of grape stems based on deep learning

Automatic recognition, segmentation, and location of grape stems'' picking points are important aspects for the picking operation of grape-picking robots. In the actual scene of an orchard, it is extremely difficult to accurately identify and segment grape stems and then locate the picking point, due to strong similarity between stems and the surrounding environment, as well as other conditions such as weather, light, and occlusion. Therefore, big challenges are posed for the grape-picking robots to perform picking operations. Recognition and the optimal picking point location of grape stem based on deep learning were proposed in this study. Considering that shape of small grape stems and their color would gradually change, a Mask Region with Convolutional Neural Network (Mask R-CNN) instance segmentation model was optimized. This model was divided into three modules, including backbone, region proposal network, and three branches. The backbone network aimed at obtaining a feature map with different levels. The regional proposal network''s target was to find regions with grape stems. And the three-branches network aimed at obtaining classification, bounding-box regression, and mask calculation of grape stems. The result of recognition and segmentation of grape stem in pixel-level was obtained through the training model, with the category and position of the grape stem were returned. To improve the segmentation effect on grape stems, the study adopted the idea of color threshold segmentation, the HSV (Hue, Saturation, Value) color space of each grape stem in the segmentation result was analyzed in segments. The average value of HSV color components of each segment was taken as the benchmark color threshold of the stem in this segment. Based on this threshold, an improved regional growth algorithm was introduced to automatically adjust and optimize the shape of the segmented grape stem. By this optimized shape, the centroid of the grape stem was calculated, the picking area was determined by the two horizontal sides of the grape stem that were closest to the centroid point, and the midpoint of this area was considered as the picking point. Approaches in this study were stated as follows. Grape stem regions of training samples were manually labeled, 600 images were selected as the training set, and 100 images as the verification set. In addition, data of the training set was enhanced by taking into consideration rotation, mirroring, blurring, and exposure operations, to improve the generalization ability of the model. A total of 3 000 training set images were generated. All the above measures contributed to the optimization of grape stem recognition and segmentation network based on the Mask R-CNN. An improved region growth algorithm was initiated to finely adjust results from multiple segments of grape stem segmentation, and the picking point was obtained based on the relationship between centroid and contour of the grape stem. The specific performance of the method under different weather and illumination image conditions was verified. A validation set consisted of 198 grape stem samples. The detection accuracy value and the location rate of the optimal picking point were taken to evaluate the models, and the detection effects before and after the model optimization were compared. Experimental results showed that detection accuracy in the optimized model reached up to 88%. Compared with the model before optimization, the model detection time was reduced by 24%. The success rate of picking point location taking the method of this study was 81.58%. And calculated picking points reached up to 99.43% within the optimal picking range which was manually set. Results showed that the proposed method recognized multiple types of grape stems under different weather and light conditions, results of grape stem region segmentation were also satisfactory. This method was capable to locate the picking point quickly and was suitable for the grape-picking robots to perform picking operations in an orchard with a complex environment. In this study, the deep learning model could be applied for the first time in the research of stem identification and segmentation and it could be an approach for grape-picking robots to pick grapes efficiently and intelligently in the natural environment.