DevOps – Make Your Cluster Hum in 10 Easy Steps

DevOps - Make Your Cluster Hum in 10 Easy Steps

Tuesday, June 19
2:50 PM - 3:30 PM
Meeting Room 230C

Managing a Hadoop cluster at scale is complex, error-prone, and tedious without the right tools. This talk will cover 10 easy steps to automate the operations of managing your cluster and ensure its health is in top shape. We will focus on DevOps tools (Chef), scripts, cron-jobs (using Apache Airflow), and dashboards (using Superset and DataDog) to automate the process for cluster operators and SREs. First, use versioned-controlled Chef roles and recipes to automatically add hosts to Ambari or Cloudera Manager, and treat configs as code and minimize changes using a local HA proxy for service discovery. Next, use advanced telemetry to monitor cluster health using Apache Superset dashboards and DataDog dashboards and monitors. We will cover how to create useful queries, slices, and dashboards in Superset, how to create meaningful metrics in DataDog, and highlight the most significant indicators of cluster health to send PagerDuty alerts. Because of the inherent failures of commodity hardware, we will also show proactive steps to automatically fix bad hosts (low disk, decommission DataNodes, etc.) by scheduling tasks in Apache Airflow. Finally, we will talk about safe deployments of configs and jars by doing rolling converges and restarts using Chef. The goal is to use data-driven metrics and scripts to make it easier to manage your cluster proactively.

SPEAKERS

Alejandro Fernandez
Infrastructure Software Engineer
Airbnb
Alejandro Fernandez has 10 years as a software engineer and 3+ years in the Hadoop ecosystem. He's currently at Airbnb in the Data Infrastructure team working on Hadoop security, Apache Airflow (incubating), and Hive. Previously, he was at Hortonworks for 3 years where he became a PMC for Apache Ambari and contributed to features like Rolling & Express upgrades. He has spoken at major conferences like Dataworks Summit in San Jose and Melbourne, and at the Apache Big Data conference in Miami. He graduated from Carnegie Mellon University, where he got his Bachelor of Science in Computer Science and additional major in Mathematics.