Data Quality Anomaly Detection and Suggestion Engine

Project Description

Summary

We are a large IT infrastructure organization looking to improve the quality of our operational infrastructure monitoring data.  Our goal with this project is to develop an API that will detect anomalies in single tables of structured data using a combination unsupervised machine learning methods and defined rules and suggest new values for anomalous data.  The API will support a larger project which includes the visualization of these anomalies in a dashboard, however this Experfy project will focus on the API and ML modelling.

Scope of Work

The selected expert will be responsible for:

•Defining an API to provide anomaly detection and suggestion services

•Developing an unsupervised learning model to detect anomalous data related to each of 24 Key Business Elements (KBEs) in our data

•Coding logic to detect additional anomalies according to predefined rules for each KBE

•Implementing an the defined API with the completed model and ML implementation

•Demonstrating the robustness of the model using various test data sets, including data with both similar (tech/infrastructure) and dissimilar (Fisher’s iris flowers, etc.) contexts

•Supporting QA and visualizer dashboard development efforts as bugs or issues are discovered in the API or model

The primary output of this project is an API for detecting anomalies in our infrastructure operations data.

The attached presentation provides additional details around the data, environment, and project requirements and gives additional context to the broader project scope (including the visualization dashboard project).  Since, this project focuses on the data anomaly detection engine only, project details out of scope for this Experfy project posting have been greyed out for scope clarity, however they are still very relevant to your implementation.

Challenge Format

We plan to hire more than one expert to implement their model using a common initial data set.  The different approaches will be evaluated after initial implementation and only one expert will be asked to continue with the project refining their model and build the API.  The period for determining which approach will be used (and who will complete the final project deliverable) will be variable but is expected to last 1-2 weeks.

Proposal

As part of your proposal please answer the following questions:

•Please describe the approach you intend to use to solve this problem (please describe both anomaly detection and value suggestion).

•What trade-offs you are making when choosing one approach over another?

•Which technology stack would you use for this challenge?

•What are the underlying assumptions you are making about the data set for this proposal?

•How would you approach in tuning the parameters for the chosen approach

•How do you plan to evaluate the performance of the model?

•How do you plan to develop the API?