The First Mile — Edge and IoT Data Collection with Apache NiFi and MiNiFi

The First Mile -- Edge and IoT Data Collection with Apache NiFi and MiNiFi

Thursday, October 11
2:00 PM - 2:40 PM
Orchard

Apache NiFi MiNiFi allows data collection in brand new environments — sensors with tiny footprints, distributed systems with intermittent or restricted bandwidth, and even disposable or ephemeral hardware. Not only can this data be prioritized and have some initial analysis performed at the edge, it can be encrypted and secured immediately.

Abstract: Apache NiFi provided a revolutionary data flow management system with a broad range of integrations with existing data production, consumption, and analysis ecosystems, all covered with robust data delivery and provenance infrastructure. Now learn about the follow-on project which expands the reach of NiFi to the edge, Apache MiNiFi. MiNiFi is a lightweight application which can be deployed on hardware orders of magnitude smaller and less powerful than the existing standard data collection platforms. With both a JVM compatible and native agent, MiNiFi allows data collection in brand new environments — sensors with tiny footprints, distributed systems with intermittent or restricted bandwidth, and even disposable or ephemeral hardware. Not only can this data be prioritized and have some initial analysis performed at the edge, it can be encrypted and secured immediately. Local governance and regulatory policies can be applied across geopolitical boundaries to conform with legal requirements. And all of this configuration can be done from central command & control using an existing NiFi with the trusted and stable UI data flow managers already love.

Expected prior knowledge / intended audience: developers and data flow managers should have a passing knowledge of Apache NiFi as a platform for routing, transforming, and delivering data through systems (a brief overview will be provided). The talk will focus on extending the data collection, routing, provenance, and governance capabilities of NiFi to IoT/edge integration via MiNiFi.

Takeaways: Attendees will learn about opportunities to bring their data flow and capture closer to the "edge" -- sources of data like IoT devices, vehicles, machinery, etc. They will understand the possibilities to prioritize, filter, secure, and manipulate this data earlier in the data lifecycle to enhance their data visibility and performance.

SPEAKERS

Andy LoPresto
Sr. Member of Technical Staff
Hortonworks
Andy LoPresto is a Sr. Member of Technical Staff at Hortonworks working on the Hortonworks DataFlow team. In this role he serves as both a Committer and Project Management Committee Member for Apache NiFi, an open source, robust, secure data routing and delivery system. Andy focuses on security concerns within NiFi including identity management, TLS negotiation, data protection, access control, encryption and hashing. Andy is also involved with the sub-project, Apache MiNiFi, which drives edge data collection, including secure command and control and immediate data provenance and governance. He has presented about NiFi at DataWorks Summits in Singapore, Tokyo, Melbourne, Berlin, Sydney, and San Jose, FOSDEM '17 in Brussels, and the OpenIoT Summit 2017.