SDLC with Apache NiFi

SDLC with Apache NiFi

Thursday, June 21
11:30 AM - 12:10 PM
Executive Ballroom 210B/F

Join us as we walk through examples of integrating Apache NiFi into existing enterprise ETL environments. We'll look at how to solve the challenges of integrating a real-time, interactive dataflow tool like NiFi into traditional ETL workflows, touching on common topics like design and deployment, version control, dataset testing, environment variables, and code promotion.

We will demonstrate how to manage changes to your NiFi data flows using SDLC approaches traditionally applied to batch processing scripts. We will walk you through the building blocks of achieving continuous integration and deployment (CI/CD) for your data flows. This includes examples of NiFi automation.

We will cover:

- Using the NiFi Command Line Interface (CLI) to script interaction with NiFi and NiFi Registry
- Utilizing NiPyApi, a Python module that provides an abstraction for interacting with the NiFi REST API, for advanced automation
- Setting up Docker-based integration test environments for NiFi

Lastly, we will demonstrate how to integrate these tools in SDLC workflows for NiFi flow versioning.

Presentation Video


Kevin Doran
Software Developer
Kevin Doran is Software Developer on the Data Flow Management team at Hortonworks. He is also a committer on the Apache NiFi project, where he contributes to the NiFi Registry flow versioning tools and the security framework for various NiFi projects. At Hortonworks, he is helping design and implement a scalable command and control solution for Apache MiNiFi deployments. Kevin is enthusiastic about open source software and building communities of developers. He enjoys automating workflows and creating simple technology that solves practical problems. Prior to joining Hortonworks, Kevin helped build a global scale optimization platform at Videology and advanced numerous research projects in the areas of cyber security, networking protocols, and secure mobile computing at BBN Technologies. Kevin holds a B.S. in Computer Engineering from the University of Maryland (go Terps!). He enjoys listening to podcasts, reading, and spending time with his two kids.