基于R-FCN深度卷积神经网络的机器人疏果前苹果目标的识别

疏果前期苹果背景复杂、光照条件变化、重叠及被遮挡,特别是果实与背景叶片颜色极为相近等因素,给其目标识别带来很大困难。为识别疏果前期的苹果目标,提出基于区域的全卷积网络(region-based fully convolutional network,R-FCN)的苹果目标识别方法。该方法在研究基于ResNet-50和ResNet-101的R-FCN结构及识别结果的基础上,改进设计了基于ResNet-44的R-FCN,以提高识别精度并简化网络。该网络主要由ResNet-44全卷积网络、区域生成网络(RegionProposal Network, RPN)及感兴趣区域(Region of Interest, RoI)子网构成。ResNet-44全卷积网络为基础网络,用以提取图像的特征,RPN根据提取的特征生成Ro I,然后Ro I子网根据ResNet-44提取的特征及RPN输出的Ro I进行苹果目标的识别与定位。对采集的图像扩容后,随机选取23 591幅图像作为训练集,4 739幅图像作为验证集,对网络进行训练及参数优化。该文提出的改进模型在332幅图像组成的测试集上的试验结果表明,该方法可有效地识别出重叠、被枝叶遮挡、模糊及表面有阴影的苹果目标,识别的召回率为85.7%,识别的准确率为95.1%,误识率为4.9%,平均速度为0.187 s/幅。通过与其他3种方法进行对比试验,该文方法比FasterR-CNN、基于ResNet-50和ResNet-101的R-FCN的F1值分别提高16.4、0.7和0.7个百分点,识别速度比基于ResNet-50和ResNet-101的R-FCN分别提高了0.010和0.041 s。该方法可实现传统方法难以实现的疏果前苹果目标的识别,也可广泛应用于其他与背景颜色相近的小目标识别中。

Recognition of apple targets before fruits thinning by robot based on R-FCN deep convolution neural network

Abstract: Before fruit thinning, factors such as complex background, various illumination conditions, foliage occlusion, fruit clustering, especially the extreme similarities between apples and background, made the recognition of small apple targets very difficult. To solve these problems, a recognition method based on region-based fully convolutional network (R-FCN) was proposed. Firstly, deep convolution neural network including ResNet-50 based R-FCN and ResNet-101 based R-FCN were studied and analyzed. After analyzing the framework of the 2 networks, it was obviously that the difference between these 2 networks was the ''conv4'' block. The ''conv4'' block of ResNet-101 based R-FCN was 51 more layers than that of ResNet-50 based R-FCN, but the recognition accuracy of the 2 networks was almost the same. By comparing the framework and recognition result of ResNet-50 based R-FCN and ResNet-101 based R-FCN, A R-FCN based on ResNet-44 was designed to improve the recognition accuracy and simplify the network. The main operation to simplify the network was to simplify the ''conv4'' block, and the ''conv4'' block of ResNet-44 based R-FCN was 6 layers less than that of ResNet-50 based R-FCN. The ResNet-44 based R-FCN consisted of ResNet-44 fully convolutional network, region proposal network (RPN) and region of interest (RoI) sub-network. ResNet-44 fully convolutional network, the backbone network of R-FCN, was used to extract features of image. The features were then used by RPN to generate RoIs. After that, the features extracted by ResNet-44 fully convolutional network and RoIs generated by RPN were used by RoI sub-network to recognize and locate small apple targets. A total of 3 165 images were captured in an experimental apple orchard in College of Horticulture, Northwest A&F University, in City of Yangling, China. After image resizing and manual annotation, 332 images, including 85 images captured under sunny direct sunlight condition, 88 images captured under sunny backlight condition, 86 images captured under cloudy direct sunlight condition, 74 images captured under cloudy backlight condition, were selected as test set, and the other 2 833 images were used to train and optimize the network. To enrich image training set, data augment, including brightness enhancement and reduction, chroma enhancement and reduction, contrast enhancement and reduction, sharpness enhancement and reduction, and adding Gaussian noise, was performed, then a total of 28 330 images were obtained with 23 591 images randomly selected as training set, and the other 4 739 images as validation set. After training, the simplified ResNet-44 based R-FCN was tested on the test set, and the experimental results indicated that the method could effectively apply to images captured under different illumination conditions. The method could recognize clustering apples, occluded apples, vague apples and apples with shadows, strong illumination and weak illumination on the surface. In addition, apples divided into parts by branched or petiole cloud also be recognized effectively. Overall, the recognition recall rate could achieve 85.7%. The recognition accuracy and false recognition rate were 95.2% and 4.9%, respectively. The average recognition time was 0.187 s per image. To further test the performance of the proposed method, the other 3 methods were compared, including Faster R-CNN, ResNet-50 based R-FCN and ResNet-101 based R-FCN. The F1 of the proposed method was increased by 16.4, 0.7 and 0.7 percentage points, respectively. The average running time of the proposed method improved by 0.010 and 0.041 s compared with that of ResNet-50 based R-FCN and ResNet-101 based R-FCN, respectively. The proposed method could achieve the recognition of small apple targets before fruits thinning which could not be realized by traditional methods. It could also be widely applied to the recognition of other small targets whose features are similar to background.