Pipeline Builder: Micron’s Journey Automating the Global Data Warehouse

Pipeline Builder: Micron’s Journey Automating the Global Data Warehouse

Wednesday, May 22
2:00 PM - 2:40 PM
Marquis Salon 13

With 12 manufacturing facilities in 5 countries, Micron has valuable business data dispersed around the world. Collecting, collating, validating and presenting this data has proven to be a significant challenge. After hand building dozens of ingestion pipelines, and considering the significant amount of pipelines remaining, it became clear that automation was the only path to success. Rising to the challenge, Micron engineers designed and developed Pipeline Builder, a Java and Angular based platform that automates the deployment of NiFi pipelines to acquire, transmit and ingest data into the Global Data Warehouse. Pipeline Builder has since been used to create and monitor hundreds of ingestion pipelines into the Global Data Warehouse. With so many concurrently active pipelines, it can be difficult to understand the health of each pipeline. Pipeline Manager and Pipeline Validator were created to track each pipeline's health and run automated validation routines to ensure data is transferred accurately from source to target.

Presentation Video


Paul Gibeault
Solution Architect, Big Data
Micron Technology Inc.
Paul Gibeault has a B.S. degree in computer science from the University of Idaho and has spent most of his career developing software frameworks and tools to enable the automation of semiconductor equipment. He is currently a Big Data Solution Architect with the IT group at the third largest memory manufacturer in the world, Micron Technology, headquartered in Boise, Idaho. For the past five years he has focused on the design and implementation of an automated approach to create Micron’s Global Data Warehouse.
Peter Wicks
Hadoop Engineer
Micron Technologies Inc.
Focused on architecting ingestion feeds through Apache NiFi and loading data into Hadoop and Teradata. In my spare time I like to work network my house and build do fun things with IoT devices.