Stream Processing System on Amazon Web Services for Gulf Gas Stations

Project Description

Gulf believes that advanced data mining techniques can be used to create a truly unique and differentiating user experience at the station, and intend to develop a system for handling the identification, payments and data intelligence for retail customers. This project is one of several which are being offered through Experfy which, in the aggregate, will represent the initial version of our customer analysis solution.

Our system has been designed to emphasize the following (in decreasing order of priority):

• security
• reliability
• speed to market
• flexibility for future enhancements

To support these objectives, the system has been designed to support both batch processing and stream processing using an architecture similar to the well known “Lambda Architecture” pattern.

In this component of the solution, we are focusing on the batch processing components of the system. The emphasis in this project is on the development of ETL processing to support both our business intelligence function as well as our promotional campaign management. This component does not contain a significant element of machine learning.

Scope of Effort

We are looking for a developer to develop the batch processing components of our system. We have developed a reference architecture consisting of a combine real-time and non-real-time processing flows (see https://www.sugarsync.com/pf/D6703166_07623628_708603) but expect that the developer will build upon this design as appropriate. Our emphasis has been to develop and deploy a fieldable system as quickly as possible, knowing that we will incorporate additional functionality as we grow. At the same time, we wish to reduce the need to maintain a large in-house IT support staff. As a result, the architecture has been designed to heavily leverage Amazon Web Services. Our expectation is that the bulk of the development associated with this portion of the architecture will incorporate some combination of AWS Kinesis, Lambda, Elastic Map Reduce, Redshift and other services for communications (e.g. SNS or SES) as well as some form of visualization / querying tool. Other projects will focus on the analytics and machine learning aspects of the project.

At a high level, this project will result in a system which accomplished the following:

• perform a regularly scheduled ETL process for collecting and transforming previously collected event data into a data warehouse suitable for supporting business intelligence functions
• manage the data components of our customer loyalty program, including the creation of notifications requesting reviews of recent customer experience
• collect event data through our RESTful API component
• disseminate these events to processing elements using a streaming system
• store and process these events to create a customer profile and support the desired in-station user experience
• a critical component of the in-station user experience is the identification of marketing campaigns relevant to the current customer
• provide various forms of notification to the customer and local sales persons
• support a combination of pre-defined and ad hoc queries against the data warehouse

The project will include the following development efforts:

• a customer facing application targeted for Android phones
• a browser based desktop application, including a tool for defining and managing promotional campaigns
• cloud based processing infrastructure
• cloud based processing infrastructure
• an appropriate business intelligence tool. This visualization and querying tool could be constructed using with existing tools (e.g. Tableau), through the AWS hosted system, QuickSite, or perhaps using an approach based upon custom development, e.g. D3.
• cloud based processing infrastructure

In addition to the user facing applications, the system being developed will also need to be integrated with the following existing, external systems:

• MAC address sniffers in the stations
• License plate readers in the stations
• Payment processing systems
• Government regulatory agency systems

More details are provided in the accompanying requirements document.

We expect that the developer for this project will need to coordinate closely with the developer of our analytics and machine learning system to ensure that the overall system satisfies the client requirements.

NOTE: If you want to take on only parts of this project, please feel free to submit a proposal.

Proposal Requirements

In your proposal, please provide:

1. previous work that you have done that is relevant;
2. how you would approach this development exercise; and
3. estimated hours and budget.