MongoDB分页技术改进与优化

随着数据量的不断增加,使用MongoDB数据库内置的skip和limit组合分页算法效率低下,成为影响数据库访问性能提升的重要问题。从分析影响分页查询速度的关键因素入手,提出细粒度查询改进算法和where-limit算法。通过理论推导和与原算法的实验结果的比较,分析了两种新算法使用场景和优缺点。将两种算法应用于实际Web2.0日志系统应用中,取得了较好的效果。最后,对影响数据分页的其它因素进行了探讨,以更好的提高Web应用性能。     

滇池流域非点源污染文档型时空数据模型的研究与应用

针对滇池流域非点源污染应急响应、模拟及数据挖掘过程中,现有的基于关系数据库建立的时空数据模型进行了分析并指出其不足,提出了基于Mongo DB及面向对象时空数据模型改进的文档型时空数据模型,以有效组织、存储与管理非点源数据;最后设计了滇池流域非点源污染时空数据管理系统,实现不同来源、格式数据的导入导出、编辑、索引及高级运算功能。试验证明,系统在存储、管理和维护海量时空数据等方面得到了提升,能为滇池流域非点源污染时空过程数据的模拟以及大数据挖掘提供统一的访问接口和数据支撑。     

Practical implications of using non-relational databases to store large genomic data files and novel phenotypes

The objective of our study was to provide practical directions on the storage of genomic information and novel phenotypes (treated here as unstructured data) using a non-relational database. The MongoDB technology was assessed for this purpose, enabling frequent data transactions involving numerous individuals under genetic evaluation. Our study investigated different genomic (Illumina Final Report, PLINK, 0125, FASTQ, and VCF formats) and phenotypic (including media files) information, using both real and simulated datasets. Advantages of our centralized database concept include the sublinear running time for queries after increasing the number of samples/markers exponentially, in addition to the comprehensive management of distinct data formats while searching for specific genomic regions. A comparison of our non-relational and generic solution, with an existing relational approach (developed for tabular data types using 2 bits to store genotypes), showed reduced importing time to handle 50M SNPs (PLINK format) achieved by the relational schema. Our experimental results also reinforce that data conversion is a costly step required to manage genomic data into both relational and non-relational database systems, and therefore, must be carefully treated for large applications.