Practice of large Hadoop cluster in China Mobile

Practice of large Hadoop cluster in China Mobile

Thursday, June 21
9:30 AM - 10:10 AM
Grand Ballroom 220B

China Mobile Limited is the leading telecommunications services provider in China, with more than 800 million active users. In China Mobile, distributed big data clusters are built by branch companies in each province for their unique requirements. Meanwhile, we have built a centralized Hadoop cluster with scale more than 1600 nodes, on which we collect data from dozens of distributed clusters and make analysis for our business.

In this session, we will introduce the architecture of the centralized Hadoop cluster and experience of constructing and tuning this large scale Hadoop cluster. Key points are as follows:
1. About Ambari: we improve Ambari with features like supporting HDFS Federation and Ambari HA , improving its performance and enabling it to support up to 1600 nodes.
2. About HDFS: we build a large HDFS cluster with data up to 60PB, using federation, ViewFS, FairCallQueue. Our best practice of cluster operation and management will also be included.
3. About Flume: We use the reformed Flume to collect data as much as 200TB per day.

Presentation Video

SPEAKERS

YUXUAN PAN
Software Engineer
China Mobile(Suzhou) Software Technology Co.,Ltd
PAN YUXUAN is a software development engineer in BigData products division , China Mobile(Suzhou) Software Technology Co. since 2014. He focus on researching and developing the CMH(China Mobile Hadoop) products. He is the Project Manager of China Mobile Centralized Business Analysis System and now he is also responsible for Data Management Suit in China Mobile.
Duan Yunfeng
Chief designer of China Mobile's big data system
China Mobile Communications Corporation
Duan Yunfeng is Chief designer of China Mobile's big data system ,and he is a Postdoctoral in the information processing Department of Peking University. Dr. Duan Yunfeng has undertaken the design, construction and operation of China Mobile's data warehouse and big data center, and has accumulated 16 years of practical experience in the field of big data. He led the team, from the system construction to the system operation and maintenance, and established a lot of big data applications. The design of technical documents, there are more than 150 copies and about 12000000 words. The data model, data interface, system architecture, quality control, business application, system security and other fields are involved in the big data system. He has published two books: Big data and big analysis and Big data Internet thinking. China Mobile big data system currently has 16 thousand nodes (X86), which exceeds 500PB system capacity, including data volume 300PB, 800 million customers' active data.