基于Solr的农田数据索引方法与大数据平台构建

针对农田数据在高吞吐量、高并发、多条件处理过程中易产生运算负载大、响应速度慢等难题,研究了负载均衡大规模集群数据处理技术,优化了多条件检索时Hbase农田数据库,提出了基于Solr的二级非主键索引方法,搭建了基于Hadoop的农田大数据平台,采用农机深松、植保、保护性耕作等8种作业生成的100TB数据对平台进行了检索实验和压力测试实验。实验结果表明,多条件检索时,优化后的技术模型在数据规模达到5×107条时,系统的响应时间小于1s,优化的性能与原生Hbase相比提高了3倍;在模拟用户达到5×105次时,系统的QPS及TPS提高了1倍左右、RT提高了2.5倍,系统的平均响应时间为183ms。本研究解决了高吞吐量、高并发导致农田数据检索效率低的问题,提高了海量农田数据实时处理的计算能力。     

Developing an Open Access Croplands Research Database Through Global Collaboration

This article describes the processes, challenges, and outcomes of a project undertaken by Kansas State University (K-State) Libraries and a global community of researchers. The project, initiated by librarians in the newly created Faculty and Graduate Services Department, involved collaboration with a K-State agronomist. The initial concept was to create an open access database of croplands research submitted by researchers from the Global Research Alliance Croplands Research Group, a consortium of over 30 countries. Due to the project's complexity, it was determined that a more manageable approach would be to pilot the project by including research from only the United States and Australia to resolve problems before scaling up to include all 34 countries in the GRA Croplands Research Group.     

基于Heritrix+Solr的农业信息垂直搜索引擎研究与设计

随着农业信息化、智能化的不断发展,农业信息量呈现井喷式增长,为广大农业从业者和农业科研人员提供便捷有效的信息检索方法是目前农业搜索引擎亟需解决的问题.为此,本文提出了基于Heritrix+Solr的农业信息垂直搜索引擎框架,并设计了适用于农业信息垂直搜索引擎的隐马尔科夫Web信息抽取模块和基于词典的mmseg4j中文分词模块,同时改进了页面排序算法,对进一步提升农业垂直搜索引擎的用户体验和工作效率具有一定的参考价值.