From docker to kubernetes: running Apache Hadoop in a cloud native way

From docker to kubernetes: running Apache Hadoop in a cloud native way

Wednesday, June 20
11:00 AM - 11:40 AM
Executive Ballroom 210B/F

Creating containers for an application is easy (even if it’s a goold old distributed application like Apache Hadoop), just a few steps of packaging.

The hard part isn't packaging: it's deploying

How can we run the containers together? How to configure them? How do the services in the containers find and talk to each other? How do you deploy and manage clusters with hundred of nodes?

Modern cloud native tools like Kubernetes or Consul/Nomad could help a lot but they could be used in different way.

It this presentation I will demonstrate multiple solutions to manage containerized clusters with different cloud-native tools including kubernetes, and docker-swarm/compose.

No matter which tools you use, the same questions of service discovery and configuration management arise. This talk will show the key elements needed to make that containerized cluster work.

Tools:
kubernetes, docker-swam, docker-compose, consul, consul-template, nomad
together with: Hadoop, Yarn, Spark, Kafka, Zookeeper, Storm….

References:
https://github.com/flokkr

Presentation Video

SPEAKERS

Márton Elek
Lead Software Engineer
Hortonworks
Have more then 15+ years of Java experiences and during theses years worked with allmost all the form of Java solutions from the low-latency multithread application to highly distributed enterprise application as developer, architect and trainer. Currently working with the Apache bigdata projects and created various type of containerized solution for the components of the Hadoop ecosystem. Founder of the first Hungarian Java User group and regular speaker at meetup events and conferences. Committer of Apache Hadoop and Apache Ratis project and working on the Apache Hadoop Ozone project and the dockerization of Apache Hadoop,