End to End Configuration Management with Puppet

This video provides an overview of the entire course.

In this video, we will talk about what configuration management actually means and how it is useful. Define configuration management, understand why you should use configuration management, & know the benefits of configuration management.

In this video, we will talk about bootstrap in the course environment. Clone our course environment GIT repository, install the Puppet client, & check that Puppet works in our environment.

In this video, we will write our first Puppet manifest and apply it with the Puppet command. Explain what a Puppet manifest is, write our first Puppet manifest, & apply our first Puppet manifest.

This video talks about how to get help from Puppet and how to get information about the current state of system resources. Explain commands for getting help from Puppet & explain how we can inspect system resources

Get to know how we can manage files, directories, and links with Puppet. Explain the file resource type, manage a simple file and directory with Puppet, & manage a symbolic link with Puppet.

You accidentally removed a file and would like to recover it. Explain Puppet file buckets, locate file buckets, & recover deleted files with the Puppet command.

You would like to install or remove software packages. Explain the Puppet package resource type, explain the package providers, & install the packages.

You would like to manage system services with Puppet. Explain the Puppet service resource type, explain the enable and ensure parameters, manage the NTP service with Puppet.

Understanding metaparameters, explain the basics of metaparameters, explain the most important metaparameters, & use the schedule metaparameters.

How can we order the application of Puppet resources? Explain the ‘before’ and ‘after’ metaparameters & create files in a specific order.

How can we notify services when a configuration file changes. Explain the ‘notify’ and ‘subscribe’ metaparameters & restart an example service on a file change.

Execute system commands with Puppet. Explain the ‘exec’ resource type, recognize why the ‘exec’ resource type is a last resort for managing resources, & run a system command with ‘exec’.

Learn how to manage cron jobs with Puppet. Explain the ‘cron’ resource type, understand ‘unmanaging’ a resource parameter, & schedule an example cron job with Puppet.

Understand Puppet modules. Explain the basics of Puppet modules, understand the module file system layout, & create an example module.

Understanding Puppet classes, explain the basics of Puppet classes and class parameters, understand the basics of Puppet data types, & create an example class using data types.

Learn how to write good Puppet modules & talk about the most common best practices when creating Puppet modules.

Learn how to install, configure, and start the NTP daemon & write a module for installing, configuring, and managing the NTP daemon.

Learn how to manage complex configuration files with Puppet, explain the basics of templates, understand the two types of templates in Puppet, & create an example template.

Learn how to manage the NTP configuration file with a template & expand the example NTP module using a template for managing the ntp.conf file.

This video provides an overview of the entire course.

As infrastructure grows to hundreds and thousands of nodes, the capacity of the Puppet server reaches a limit. Load balancing in multiple Puppet servers is an optimal way to handle load and provide redundancy, understand a highly available and scalable Puppet architecture, configure Puppet servers to work with load balancers, & Apache HTTP server as a load balancer for Puppet service.

As infrastructure grows, isolating the Puppet server components is a better way to manage them, rather than having a monolithic Puppet server node. Let’s take a look at isolating and centralizing the Puppet certificate authority service, which is an important step in this process. Understand the split CA architecture, create and configure the Puppet CA node, & Integrate the Puppet CA node with the load balanced infrastructure.

With the growth in the number of agent nodes, maintaining a database and the history of node configuration and metadata is crucial for any infrastructure. PuppetDB helps to maintain this database and also aids in configuration via exported resources. Install and configure the PostgreSQL database, the PuppetDB, and connect it to the PostgreSQL database, & integrate PuppetDB with Puppet server.

Infrastructure management involves services with server and client based models. The server has to push or pull the data from various nodes in the infrastructure for which it needs the node specific information. To automate this process, the methods of exporting and collecting in Puppet modules are extremely useful. Define virtual or exported resources in Puppet client modules, send exported resources information to PuppetDB, & get exported resources data from PuppetDB and generate configuration using server modules.

After the data from thousands of nodes are stored in a database, they are not very useful, unless they can be queried and visualized using structured querying mechanisms. Different Puppet query mechanisms and dashboards helps in solving these problems. Configure workstation, and understand methods to query PuppetDB, understand PuppetDB API endpoints and perform queries, & use default PuppetDB dashboard for visualization.

As the number of nodes and service roles grow in an infrastructure, it is important to have proper node classification mechanisms so that nodes are able to receive catalogs correctly and without latency. Proper manageable node classification methods are also desirable. Understand external node classification concept, create scripts to provide node definition and classification data, & configure Puppet server to use scripts as source of node classification data.

Hostname based node classification often gets very complex and it is easier to switch to other node classification methods. Apart from providing data isolation from code, Hiera can also be an effective framework to provide node classification data based on metadata such as facts. Understand Hiera file structure for node classification, configure Hiera for node classification, & populate Hiera data file with fact specific data and module information.

Running Puppet components in Docker containers is a bit different than doing it on a physical or virtual machine. It is important to understand how the service components work in an isolated manner and also in the form of an application stack. Understand containerized service deployment method and configure Docker on workstation, configure and run Puppet server in Docker container & Puppet server side components using Docker Compose.

Running Puppet agent in Docker Containers to automate them is different from the general process; given the fact that container based dynamic environments don’t have hostnames for the containers and system based process control systems are unavailable in containers. Create a Docker file for Puppet agent, run Docker containers with Puppet agent, & use Hiera to fetch service specific catalog data.

The fact that containers are able to be spawned within few seconds is true. This is due to the fact that Docker images already have all the configurations baked in them and provisioning during the container launch is not required, thus bringing down the deployment time. Creating images is a crucial step in container based environments. Build and push image manually, with dependencies using image_build Puppet module, & use Packer to build, tag and push Docker images.

Once Puppet server side components and containers are ready to be spawned using Docker images, it is important that we are able to launch application stacks involving multiple components such as application and database. Docker Compose can be managed efficiently by use of respective Puppet modules and help us in managing container based application stacks. Understand community modules to manage Docker, create Puppet module to configure Docker environment & deploy application stacks using Docker Compose.

Once infrastructure reaches a considerable size, a requirement is to be able to perform specific tasks on specific nodes. Managing and maintaining an inventory of role based nodes is a tedious process. MCollective can be a very efficient tool in implementing orchestration which helps us to target nodes based on metadata and run specific tasks. Understand Orchestration and various components of the framework, configure ActiveMQ message broker as the middleware & MCollective servers and clients using Puppet modules.

Although facter provides a lot of system facts, often in complex environments, they are not enough. By writing custom facts based on our infrastructure, we can add meaningful custom data to our infrastructure nodes based on which complex tasks can be performed. Understand custom fact concept and directory structure, write custom fact, & make facts dependent on system or other custom facts.

Puppet built-in types are very helpful in writing configurations. However, for simplified use of Puppet modules, often writing custom types are very important. For users who don’t wish to get into the complex methods of writing Ruby based Puppet types, resource types are a simpler alternative where custom types can be written using the Puppet language. Learn concept of resource type, write custom resource type using ‘define’ method, & apply resource type to agent nodes.

With the numerous tools and services available in the infrastructure ecosystem, it is quite exhausting and time consuming to write Puppet modules for them from scratch using built-in Puppet types. For such scenarios, advanced Puppet types can be written using Ruby which can simplify custom tasks. Understand custom Puppet type concept and directory structure, create custom type and understand options, & apply custom Puppet type to agent nodes.

Puppet types are useful in defining custom tasks and processes, however without Providers, Puppet types are unable to do much. If Puppet types are the interface to create custom configurations, providers are the backend logic which actually performs the tasks on the systems. Multiple providers can be written for systems or services with different behaviors. Understand custom provider concept and directory structure, create custom provider and add functionality based on Puppet type. Apply custom provider to agent nodes

Although Puppet types and Providers help us in creating advanced tasks and interfaces, for some smaller tasks and scenarios, it is enough to create functions. Custom functions can be created in the Puppet language and Ruby to handle such scenarios. Understand custom function concept and directory structure, create custom function and use it in Puppet manifest, apply manifest to agent nodes.