Hadoop: The Unintended Benefits

Wednesday, February 6
11:50 AM - 12:30 PM
Room 104

The Department of Home Affairs’ initial big data use-cases were standard Hadoop fare: legacy system archival, log data retention and a scalable BLOB store. Along the way we discovered that a platform as open as HDP offered opportunities beyond those exposed by everyone’s favourite asparagus diagram. Here you’ll see how we’ve augmented our Hadoop stack with Solr for text searching, offloading significant work from our Teradata Warehouse in the process; and how we are currently implementing JanusGraph, improving the quality of models developed by our Data Scientists and providing richer data to our Intelligence Analysts – all backed by HDP!


Steven O'Neill
Director EDW Platforms, Data Warehouse
Australian Government, Department of Home Affairs
Steven O'Neill began his career as a DBA and has 20 years' experience specialising in Oracle and SQL Server. He started working with Hadoop, using Hortonworks HDP, almost four years ago. As technical lead for Hadoop within the Department of Home Affairs he has overseen the implementation of multiple projects involving data archiving, image storage and text indexing. Steven and his team are currently focusing on analytic workloads, graph storage and analysis, and improvements to streaming ingestion of data into both Hadoop and the Department's Teradata Enterprise Data Warehouse.
Dwane Hall
Hadoop Developer
Australian Government, Department of Home Affairs
Dwane Hall began his career as a Java Developer before moving into a Data Engineering role all within the Department of Home Affairs. He has been working with Hadoop and the Hortonworks HDP for the past two years. As a Hadoop Developer his focus has been on architecting and securing the Departments NiFi and Solr environments and identifying and implementing solutions that utilise the Departments data holdings to support operational decision making.