基于网络爬虫技术的健康医疗大数据采集整理系统

doi:10.6040/j.issn.1671-7554.0.2017.365

山东大学学报（医学版） ›› 2017, Vol. 55 ›› Issue (6): 47-55.doi: 10.6040/j.issn.1671-7554.0.2017.365

基于网络爬虫技术的健康医疗大数据采集整理系统

卞伟玮^1,2,王永超^2,3,崔立真^2,4,郭伟^2,4,李晖^2,4,周苗^1,2,薛付忠^1,2,刘静^1,2

1.山东大学公共卫生学院生物统计学系, 山东济南 250012;2.山东大学齐鲁生物医学大数据研究中心, 山东济南 250012;3.康评健康医疗大数据科技有限公司, 山东济南 250101;4.山东大学计算机科学与技术学院, 山东济南 250101

收稿日期:2017-04-27 出版日期:2017-06-10 发布日期:2017-06-10
通讯作者: 刘静. E-mail:liujing@sdu.edu.cn E-mail:liujing@sdu.edu.cn
基金资助:
国家自然科学基金(81273177)

A collecting and processing system for health care big data based on web crawler technology

BIAN Weiwei^1,2, WANG Yongchao^2,3, CUI Lizhen^2,4, GUO Wei^2,4, LI Hui^2,4, ZHOU Miao^1,2, XUE Fuzhong^1,2, LIU Jing^1,2

1. Department of Biostatistics, School of Public Health, Shandong University, Jinan 250012, Shandong, China;
2. Cheeloo Research Center for Biomedical Big Data, Shandong University, Jinan 250012, Shandong, China;
3. Kangping Health Care Big Data Technology Company Limited, Jinan 250101, Shandong, China;
4. School of Computer Science and Technology, Shandong University, Jinan 250101, Shandong, China

Received:2017-04-27 Online:2017-06-10 Published:2017-06-10

摘要/Abstract

摘要： 目的快速、准确地获得公共卫生服务系统的医疗数据,并进行数据整理,为建立人群健康风险评估模型提供数据基础。方法运用聚焦网络爬虫技术,设计算法并编程,在自动记录和修正URL异常、原始数据存档、保持登录方式3个方面进行算法改进。将设计好的爬虫应用于爬取已获得授权网站的医疗数据,通过医学数据库系统,对数据进行解析、整理与导出。结果获得多个公共卫生服务基地数据,为当地政府部门提供数据分析报告,利用整理分析的数据完成多项健康风险评估模型建立。结论基于网络爬虫技术建立的数据采集整理系统,可以解决获取及整理网络许可数据的难题,将此技术应用于医药卫生领域,可使现有丰富的医学数据资源得以充分利用并提高利用效率。

关键词: 聚焦爬虫, 数据采集, 数据解析, 数据整理, 网络爬虫, 数据库系统

Abstract: Objective To collect and process the medical data from public health service system rapidly and exactly, and to provide data base for establishing the population health risk assessment model. Methods The algorithm and program were based on focused web crawler. This study mainly improved the algorithm in three aspects: automatic recording and correcting URL anomaly, original data archiving and keeping login mode. Medical data of the authorized website were obtained by the advanced web crawler, and were parsed and sorted out via medical database system. Results Data from several public health service base were acquired to provide data analysis report for local government, and multiple health risk assessment models were constructed by means of the processed data. Conclusion Utilizing the data collecting and processing system based on web crawler,we can deal with the problem that acquiring and organizing the available data in real life. This technology can be applied in medicine and health field,which will make full use of the existing rich medical data resources and greatly improve the utilization efficiency.

Key words: Web crawler, Data parsing, Database system, Focused web crawler, Data collecting, Data processing

中图分类号:

R319

卞伟玮,王永超,崔立真,郭伟,李晖,周苗,薛付忠,刘静. 基于网络爬虫技术的健康医疗大数据采集整理系统[J]. 山东大学学报（医学版）, 2017, 55(6): 47-55.

BIAN Weiwei, WANG Yongchao, CUI Lizhen, GUO Wei, LI Hui, ZHOU Miao, XUE Fuzhong, LIU Jing. A collecting and processing system for health care big data based on web crawler technology[J]. JOURNAL OF SHANDONG UNIVERSITY (HEALTH SCIENCES), 2017, 55(6): 47-55.

参考文献

[1] 赵屹,卜德超.当生物医学遇上大数据[J].北大商业评论,2015(3): 74-79.
[2] 秦文哲,陈进,董力.大数据背景下医学数据挖掘的研究进展及应用[J].中国胸心血管外科临床杂志,2016, 23(1): 55-60. QIN Wenzhe, CHEN Jin, DONG Li. Progress and application of medical data mining under the background of big data[J]. Chin J Clin Thorac Cardiov Surg, 2016, 23(1):55-60.
[3] 陈锐,冯占英.大数据时代医学专业图书馆面临的挑战与对策[J].中华医学图书情报杂志,2014, 23(1): 2-6. CHEN Rui, FENG Zhanying. Challenges to medical libraries in big data era and their countermeasures[J]. Chin J Med LibrInf Sci, 2014, 23(1): 2-6.
[4] 李惠先,封二英.大数据时代医学研究面临的机遇与挑战[J].计算机光盘软件与应用, 2014(23): 138-139.
[5] 李娟.医疗卫生信息化综合大数据平台关键技术探究[J].金陵科技学院学报,2014, 30(4): 21-24. LI Juan. The key technology on integrated big data platform for informatizationof medicareand health[J]. Journal of Jinling Institute of Technology, 2014, 30(4): 21-24.
[6] 孙立伟,何国辉,吴礼发.网络爬虫技术的研究[J].电脑知识与技术,2010, 6(15): 4112-4115. SUN Liwei, HE Guohui, WU Lifa. Research on the Web Crawler[J]. Computer Knowledge and Technology, 2010, 6(15): 4112-4115.
[7] 周立柱,林玲.聚焦爬虫技术研究综述[J].计算机应用,2005, 25(9): 1965-1969. ZHOU Lizhu, LINLing. Survey on the research of focused craw ling technique[J]. Computer Applications, 2005, 25(9): 1965-1969.
[8] 于怀宝.面向建材信息的网络爬虫系统的设计与实现[D].北京:北京交通大学, 2015.
[9] 曾伟辉.支持AJAX的网络爬虫系统设计与实现[D].合肥:中国科学技术大学, 2009.
[10] 曾伟辉,李淼.深层网络爬虫研究[J].计算机系统应用,2008(5): 122-126.
[11] Chen Z, Ma J, Lei J, et al. A cross-language focused crawling algorithm based on multiple relevance prediction strategies[J]. Comput Math Appl, 2009, 57(6): 1057-1072.
[12] Batsakis S, Petrakis EGM, Milios E. Improving the performance of focused web crawlers[J]. Data & Knowledge Engineering, 2009, 68(10): 1001-1013.
[13] 杨定中,赵刚,王泰.网络爬虫在Web信息搜索与数据挖掘中应用[J].计算机工程与设计,2009, 30(24): 5658-5662. YANG Dingzhong, ZHAO Gang, WANG Tai. Application of Web Crawler in information search and data mining[J].Computer Engineering and Design, 2009, 30(24): 5658-5662.
[14] 罗一纾.微博爬虫的相关技术研究[D].哈尔滨:哈尔滨工业大学, 2013.
[15] 徐远超,刘江华,刘丽珍, 等.基于Web的网络爬虫的设计与实现[J].微计算机信息,2007, 23(7-3): 119-121. XU Yuanchao, LIUJianghua, LIU Lizhen, et al. Design and implementation of spider on web- based full-text search engine[J]. Microcomputer Information, 2007, 23(7-3): 119-121.
[16] 许笑, 张伟哲,张宏莉, 等.广域网分布式Web爬虫[J].软件学报,2010, 21(5): 1067-1082. XU Xiao, ZHANG Weizhe, ZHANG Hongli, et al. WAN-based distributed web crawling[J]. Journal of Software, 2010, 21(5): 1067-1082.
[17] Lawrence S, Giles CL. Accessibility of information on the web[J]. Nature, 1999, 400(1): 107-109.
[18] Aggarwal CC. Collaborative crawling: Mining user experiences for topical resource discovery[C] // Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining. New York: ACM, 2002: 423-428.
[19] Ahlers D, Boll S. Adaptive geospatially focused crawling[C] // ACM Conference on Information and Knowledge Management, New York: ACM, 2009: 445-454.
[20] Xu S, Yoon HJ, Tourassi G. A user-oriented web crawler for selectively acquiring online content in e-health research[J]. Bioinformatics, 2014, 30(1): 104-114.
[21] 孔抗美,张莹,李韶斌, 等.医院医疗数据挖掘与分析[J].中国卫生信息管理杂志, 2015, 8(6): 29-33.
[22] 李雄伟.数据挖掘在医疗中的应用研究[J].信息化纵横, 2009, 16: 78-82. LI Xiongwei. Application research of data mining in medical treatment[J]. Informationization, 2009, 16: 78-82.
[23] 蒋良孝,蔡之华.医疗数据挖掘及其应用[J].微型机与应用,2003, 10: 45-47.
[24] 龚卫宁.数据挖掘在医院管理中的应用[J].中国医药指南,2012, 10(12): 722-725.
[25] 李俊.数据挖掘技术在医疗信息系统中的研究与应用[D].成都:成都理工大学, 2011.
[26] 李怀庆,张文东.数据挖掘技术在医院信息系统中的应用[J].医疗设备信息, 2007, 22(12):48-49. LI Huaiqing, ZHANG Wendong. Application of data mining technology in hospital informationsystem[J]. Information of Medical Equipment, 2007, 22(12): 48-49.

多维度评价

Viewed

Full text

Abstract

Cited

Shared

Discussed

基于网络爬虫技术的健康医疗大数据采集整理系统

A collecting and processing system for health care big data based on web crawler technology

PDF (PC)

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 2

多维度评价

本文评价

推荐阅读 0

[1]	吴铭生,王苹,林俊豪,王谦,吴树明 . ERCC1基因蛋白表达与非小细胞肺癌顺铂辅助化疗的关系[J]. 山东大学学报(医学版), 2008, 46(6): 556-560.
[2]	. 靶向rPTTG的shRNA慢病毒载体构建及沉默效率评价[J]. 山东大学学报(医学版), 2009, 47(9): 76-80.