Adopting Hadoop for the Enterprise: From Strategy to Roadmap

This module describes the objectives of this course along with the background expected of the student.

A brief synopsis of the two main topics of Module 2 - Overview of Hadoop This module provides the necessary background to understand why a strategic approach is necessary to adopt Hadoop for enterprise wide use.

This lecture describes Hadoop at its highest level as being comprised of two primary value creation capabilities: storing large file and processing them in parallel. Through analogies to similar, but more traditional, technology this lecture gives the student an understanding of Hadoop to enable them to predict the benefits of its use.

This lecture contrasts using Hadoop for isolated, individual projects to using it broadly for many teams and projects. These two alternatives, called Local and Enterprise Adoption respectively, provide the basis for the challenges that can be encountered. Overcoming these challenges and arriving at a successful use of Hadoop across many teams, projects and usage categories it the motivation behind the strategic framework and architectural patterns that form the heart of this course.

Check your understanding of some of the key capabilities and challenges of Hadoop.

This lecture covers the components of the strategic framework that is used to construct a long-range roadmap for adopting Hadoop. Beginning with the target of the roadmap -- an enterprise data lake, I describe the “lanes” that contain the activities of the roadmap that drive you there along with the mile markers along the way to confirm your progress and correct course.

Diving into the business outcome lane of the roadmap, in this lecture we discuss the difference between cost-saving and business insight creation opportunities for Hadoop. Some of these opportunities are more focused on IT efficiency; while others are targeted at achieving new business analysis capabilities. As a result, we also discuss the differences in IT processes that support achieving these business outcomes.

Having described the business outcomes that should be sought as a result of adopting Hadoop for wide spread use, this lecture describes the technical components of the platform in greater detail in preparation for high-level architecture discussions in the subsequent modules. It also describes the IT process implications of Hadoop being part of your infrastructure.

Cross-cutting concerns are those topics that affect both lanes of activity in the roadmap. These topics include organizing information and processing in Hadoop, addressing security needs, maintaining metadata, governing use and capturing metrics.

Check your understanding of the concepts of the strategic framework.

In this introductory discussion about creating a value-oriented roadmap we discuss again the six use-case categories along with the kinds of value they are intended to create. These categories provide a related set of usage goals for the Hadoop platform that combine to create more value than each could on its own. Each lecture in this module covers one of the use case categories either related to cost-savings or business insight creation. For each use case, we review a typical architecture for achieving this goal without using Hadoop and the pain points often associated with doing so. After that a high-level architecture using Hadoop technologies is described along with the way it addresses the pain points and achieves the related business value. Then each architecture description is completed with a list of the technical capabilities necessary to achieve it. In this way, you can see how the technical capabilities are highly reusable from use case to use case.

For the accessible archive use case, this lecture describes * The typical architecture used without Hadoop * Pain points of that approach * A high-level Hadoop-centric architecture * How the pain points are addressed * Technical capabilities necessary to achieve it

For the processing offload use case, this lecture describes * The typical architecture used without Hadoop * Pain points of that approach * A high-level Hadoop-centric architecture * How the pain points are addressed * Technical capabilities necessary to achieve it

For the enterprise data warehouse optimization use case, this lecture describes * The typical architecture used without Hadoop * Pain points of that approach * A high-level Hadoop-centric architecture * How the pain points are addressed * Technical capabilities necessary to achieve it

Check your understanding of the critical issues of the cost-savings opportunities.

For the data exploration use case, this lecture describes * The typical architecture used without Hadoop * Pain points of that approach * A high-level Hadoop-centric architecture * How the pain points are addressed * Technical capabilities necessary to achieve it

For the 360 degree view use case, this lecture describes * The typical architecture used without Hadoop * Pain points of that approach * A high-level Hadoop-centric architecture * How the pain points are addressed * Technical capabilities necessary to achieve it

For the advanced analytics use case, this lecture describes * The typical architecture used without Hadoop ( Pain points of that approach * A high-level Hadoop-centric architecture * How the pain points are addressed * Technical capabilities necessary to achieve it

This lecture discusses a handful of technical capabilities that are critical to succeeding with the use case patterns we describe in prior lectures. These are discussed in a manner that is independent of the technical components so you can use them as a way to measure progress in gaining technical skill with the platform.

This lecture identifies a set of milestones that can be used to chart your progress with Hadoop. Grouping these milestones into categories provides you with a method by which you can grade your achievements in a manageable number of areas. These areas include ingesting data to Hadoop, processing data in Hadoop, delivering data for use, operating the platform, and supporting various types of work.

Check your understanding of the issues and concepts of the business-insight opportunities.

In this lecture, we turn from the technical to the people related aspects of adopting Hadoop. Creating a team to achieve business value is of primary importance and is discussed in this module along three primary themes: What constitutes the “right” team, how do you get into a rhythm, and how do you track your progress..

In this lecture we review the kinds of people you need on your team to be successful at creating business value, whether this is related to cost-savings or business insights. The emphasis on this team is having the people you need to find the right opportunities to pursue and demonstrate their success.

Creating a cadence involves getting into a rhythm of achieving business value through the use of Hadoop. Rhythm creates growing confidence inside and outside the team and can turn the challenge of using new technologies into a virtuous cycle and accelerated adoption.

As with any road trip, you need to know how much progress you’re making. This lecture describes some specifics about tracking your progress on the value-creation half of the roadmap framework along with some warning signs that you might be off course.

Check your understanding of organizing the checking your progress on the business track.

In this lecture, we expand upon the people related aspects of adopting Hadoop by diving into the technical R&D capabilities your team must master. It is important to create a team that continues to learn and keep pace with developments in Hadoop and related technologies. As in the last module, we discuss this along three themes: What constitutes the “right” team, how do you get into a rhythm, and how do you track your progress..

Creating a cadence involves getting into a rhythm of learning and demonstrating mastery of more and more aspects of the Hadoop platform. Rhythm creates growing confidence inside and outside the team and can turn the feeling of being overwhelmed by too much to learn into a track record of success.

As with any road trip, you need to know how much progress you’re making. This lecture describes some specifics about tracking your progress on the technology learning half of the roadmap framework along with some warning signs that you might be off course.

Check your understanding of the key concepts related organizing for the technical capabilities track.

We wrap up this course on adopting Hadoop with a lecture called “Decisions, Levers and Gates”. Decisions steer where you head. Levers affect your pace, the size of the problems you can seek to solve, and the cost of the overall platform. Gates give you places to check the work and set up governance.

Now that we’ve covered so much ground, this lecture finishes up with a bird's-eye view of the path we’ve taken and puts many of the tools and guides discussed in the course into a single place.