Sherlock: an anomaly detection service on top of Druid

Sherlock: an anomaly detection service on top of Druid

Tuesday, June 19
11:00 AM - 11:40 AM
Executive Ballroom 210D/H

Sherlock is an anomaly detection service built on top of Druid. It leverages EGADS (Extensible Generic Anomaly Detection System; github.com/yahoo/egads) to detect anomalies in time-series data. Users can schedule jobs on an hourly, daily, weekly, or monthly basis, view anomaly reports from Sherlock's interface, or receive them via email.

Sherlock has four major components: timeseries generation, EGADS anomaly detection, Redis backend and Spark Java UI. Timeseries generation involves building, validating, querying, parsing the Druid query. Parsed Druid response is then fed to EGADS anomaly detection component which detects and generates the anomaly reports for each input time-series data. Sherlock uses Redis backend to store jobs metadata, generated anomaly reports and persistent job queue for scheduling jobs, etc. Users can choose to have a clustered Redis or standalone Redis. Sherlock provides user interface built with Spark Java. The UI enables users to submit instant anomaly analysis, create, and launch detection jobs, view anomalies on a heatmap and on a graph.

SPEAKERS

Jigarkumar Patel
Software Development Engineer I
Oath Inc.
Work: Yahoo Inc/ Oath Inc (Feb 2017 to present) Intern, Yahoo Inc (June 2016 to Sep 2016) Cognizant Technology Solutions (July 2014 to May 2015) Education: University of California, San Diego (Sep 2015 to Dec 2016) Master in Computer Science Institute of Technology, Nirma University (Aug 2010 to June 2014) Bachelor in Computer Science & Engineering
David Servose
Software Systems Engineer
Oath
I graduated with a Bachelor's Degree in Computer Engineering from the University of Illinois in Champaign Urbana, during which I did 2 internships at Yahoo. After graduating, I joined Yahoo full-time and have been working on large scale batch processing systems, mostly Hadoop, and data analytics platforms.