Acquisition of Seismic, Hydroacoustic, and Infrasonic Data with Apache NiFi and Apache Accumulo

Acquisition of Seismic, Hydroacoustic, and Infrasonic Data with Apache NiFi and Apache Accumulo

Wednesday, June 20
2:00 PM - 2:40 PM
Meeting Room 230B

Hadoop Distributed File System (HDFS) based architectures allow faster ingestion and processing of larger quantities of time series data than presently possible in current seismic, hydroacoustic, and infrasonic (SHI) analysis platforms. We have developed a data acquisition and signal analysis system using Hadoop, Accumulo, and NiFi. The data model allows individual waveform samples and their associated metadata to be stored in Accumulo. This is a significant departure from traditional storage practices, where continuous waveform segments are stored with their associated metadata as a single entity. Our design allows for rapid table scans of large data archives within Accumulo for locating, retrieving, and analyzing specific waveform segments directly. The scalability of Hadoop permits the system to accommodate the ingestion and analysis of new data as a sensor network grows. Our system is currently acquiring data from over 200 SHI sensors. Peak ingest rates are approaching 500k entries per second, while preserving constant sub-second access times to any range of entries. The average load produced by the data ingest process is consuming less than 10 percent of available system resources.

Presentation Video

SPEAKERS

Charles Houchin
Computer Scientist
Air Force Technical Applications Center (AFTAC)
Charles Houchin is a computer scientist at the Air Force Technical Applications Center (AFTAC). His experience includes developing systems for government and military customers ranging from enterprise to mobile applications. Most recently, he has created data acquisition and analysis solutions involving technologies from the Hadoop ecosystem, including Apache Accumulo, Apache NiFi, and Apache Spark. William N. Junek, Charles A. Houchin, Joseph A. Wehlen, John E. Highcock, Marcus Waineo; Acquisition of Seismic, Hydroacoustic, and Infrasonic Data with Hadoop and Accumulo. Seismological Research Letters ; 88 (6): 1553–1559. doi: https://doi.org/10.1785/0220170056
John Highcock
Solutions Architect
Cloudera
John Highcock is a Solutions Architect at Cloudera (formerly Hortonworks). Prior to joining, he worked on big data projects at the US Department of Justice.