Dhalion: towards self-regulating stream processing

Dhalion: towards self-regulating stream processing

Thursday, June 21
11:30 AM - 12:10 PM
Meeting Room 230B

In recent years, there has been an explosion of large-scale real-time analytics needs, and a plethora of streaming systems have been developed to support such applications. These systems are able to continue stream processing even when faced with hardware and software failures. However, these systems do not address some crucial challenges facing their operators: the manual, time-consuming, and error-prone tasks of tuning various configuration knobs to achieve service level objectives (SLO) as well as the maintenance of SLOs in the face of sudden, unpredictable load variation and hardware or software performance degradation.

In this talk, we introduce the notion of self-regulating streaming systems and the key properties that they must satisfy. We then present the design and evaluation of Dhalion, a system that provides self-regulation capabilities to underlying streaming systems.

We describe our implementation of the Dhalion framework on top of Twitter Heron, as well as a number of policies that automatically reconfigure Heron topologies to meet throughput SLOs, scaling resource consumption up and down as needed. We experimentally evaluate our Dhalion policies in a cloud environment and demonstrate their effectiveness. We are in the process of open-sourcing our Dhalion policies as part of the Heron project.

Presentation Video

SPEAKERS

Avrilia Floratou
Senior Scientist
Microsoft
Avrilia is a Senior Scientist at Microsoft's Cloud and Information Services Lab, where her research is focused on scalable real-time stream processing systems. She is also an active contributor to Heron, collaborating with Twitter. Prior to her current role, she was a research scientist in IBM Research working on SQL-on-Hadoop systems. She holds a PhD in data management from University of Wisconsin-Madison.
Ashvin Agrawal
Senior Research Engineer
Microsoft
Ashvin Agrawal is a Senior Research Engineer at Microsoft, where he works on streaming systems and contributes to the Apache Heron and Dhalion project. Ashvin has more than 15+ years of software development experience. He specializes in developing large-scale distributed systems. Previously, he worked at VMware, Yahoo, and Mojo Networks. Ashvin holds an M.Tech. in Computer Science from IIT Kanpur, India.