Rich placement constraints: Who said YARN cannot schedule services?

Rich placement constraints: Who said YARN cannot schedule services?

Wednesday, June 20
11:50 AM - 12:30 PM
Executive Ballroom 210C/G

The rise in popularity of machine learning, streaming, and latency-sensitive online applications in shared production clusters has raised new challenges for cluster schedulers. To optimize their performance and resilience, these applications require precise control of their placements by means of complex constraints. Examples of such scenarios are the following:
• Deep learning applications need to run on GPU machines with specific GPU models and driver/kernel versions.
• Hive or Spark applications benefit from being collocated on the same rack to reduce network cost and thus speed up their execution. At the same time, it is desirable to limit the number of allocations per machine to minimize resource interference.
• Low-latency services such as HBase need to be allocated across failure domains to improve their availability.
• A DNS service might need to run on machines with public IP address.

In this talk we present the brand new addition of expressive placement constraints in YARN. We show how applications can leverage such constraints to achieve complex placements, such as collocating their allocations on the same node/rack (affinity), spreading their allocations across nodes/racks (anti-affinity), or allowing up to a specific number of allocations per node group (cardinality) to strike a balance between the two. We describe real use cases from production clusters and show the benefits of placement constraints on large clusters using popular applications in both on-prem and cloud settings.


Konstantinos Karanasos
Senior Scientist
Konstantinos Karanasos is a Senior Scientist at the Cloud and Information Services Lab (CISL) at Microsoft (based at the Silicon Valley office) and a PMC member of Apache Hadoop. His work at Microsoft has focused on resource management for the company's production analytics clusters and on query optimization for large-scale analytics. Within Apache Hadoop, Konstantinos has worked on adding support to YARN for opportunistic containers and for rich placement constraints. Prior to joining Microsoft, he was a postdoctoral researcher at IBM Almaden Research Center, where he was member of the Big Data analytics group, working on problems related to query optimization. Konstantinos obtained his PhD from Inria and the University Paris-Sud, France. In the context of his PhD, he worked in the areas of view-based query processing and semi-structured data management. He also holds a Diploma in Electrical and Computer Engineering from the National Technical University of Athens, Greece.
Wangda Tan
Engineering Manager, YARN
Wangda Tan is Product Management Committee (PMC) member of Apache Hadoop and engineering manager of YARN team at Hortonworks. His major working field is Hadoop YARN GPU isolation and resource scheduler, participated features like node labeling, resource preemption, container resizing etc. Before join Hortonworks, he was working at Pivotal, working on integration OpenMPI/GraphLab with Hadoop YARN. Before that, he was working at Alibaba cloud computing, participated creating a large scale machine learning, matrix and statistics computation platform using Map-Reduce and MPI.