High throughput data replication over RAFT

High throughput data replication over RAFT

Tuesday, June 19
4:50 PM - 5:30 PM
Executive Ballroom 210C/G

Raft protocol has been successfully used for consistent metadata replication; however, using it for data replication poses unique challenges. Apache Ratis is a RAFT implementation targeted at high throughput data replication problems. Apache Ratis is being successfully used as a consensus protocol for data stored in Ozone (object store) and Quadra(block device) to provide data throughput that saturates the network links and disk bandwidths.

Pluggable nature of Ratis renders it useful for multiple use cases including high availability, data or metadata replication, and ensuring consistency semantics.

This talk presents the design challenges to achieve high throughput and how Apache Ratis addresses them. We talk about specific optimizations that have been implemented to minimize overheads and scale up the throughput while maintaining correctness of the consistency protocol. The talk also explains how systems like Ozone take advantage of Ratis’s implementation choices to achieve scale. We will discuss the current performance numbers and also future optimizations.

SPEAKERS

Mukul Kumar Singh
Staff Software Engineer
Hortonworks
Mukul is currently working with Hortonworks and is an active contributor to Apache Hadoop and Apache Ratis project. He received his master degree from Carnegie Mellon University and bachelors from Visveswaraya Technological University. He has been working actively on filesystems for last 8 years and has worked extensively on Ozone object store, Flash based filesystems, Shingled Magnetic Recording drives, data replication and disaster recovery solutions.
Lokesh Jain
Software Engineer
Hortonworks
Lokesh Jain is a software engineer at Hortonworks. He has completed B.E.(Hons.) Computer Science and M.Sc.(Hons.) Mathematics from BITS Pilani. He is one of the early developers of Apache Ratis project and also contributes to Apache Hadoop. He also worked on GSOC project for SageMath organisation in 2017.