Extending Chef Badge

Course

November 12th, 2018

Intro Video

Photo of Keith Thompson

Keith Thompson

DevOps Training Architect II in Content

Length

16:02:44

Difficulty

Intermediate

Course Details

In this course, you will gain the ability to customize and extend Chef. Following along with the lessons and learning activities in this course will provide you with the experience necessary to be able to add features to Chef and the associated tools. For the Extending Chef Badge exam, you'll need to be able to describe how and why to extend Chef's various tools. You will also need to demonstrate that you have the ability to extend Chef. This course will provide you with all of the knowledge and hands-on experience necessary to be comfortable extending Chef and to pass the exam.

Syllabus

Course Introduction

Getting Started

Course Introduction

00:00:49

Lesson Description:

IMPORTANT: Don't forget to use the code LINUXACADEMY10 to get a discount when registering for the exam. Chef and the tools in the Chef ecosystem are great for configuration management by default, but as you continue to work with Chef it is common to dig deeper to extend what Chef already provides. This lesson gives an overview of the topics that we'll cover as we learn how to extend the built-in functionality of Chef. By the time you're finished with this course, you'll be ready to attain the Extending Chef Badge and feel comfortable customizing Chef in your real-world environment.

About a Course Author

00:00:44

Lesson Description:

A little about me, Keith Thompson.

Course Features and Tools

00:06:31

Lesson Description:

It's important for you to understand the tools and resources available to you as a Linux Academy student. This video will cover the course features used by this course and other tools you have at your disposal, such as flash cards and course schedules.

Setting Up Our Learning Environment

00:15:02

Lesson Description:

Before we start working on extending Chef we'll want to make sure that we have a Chef Server, workstation, and node to deploy to as we are testing things out. Documentation For This Video Chef Server DownloadChef Server Installation DocumentationChefDK DownloadChefDK Installation DocumentationBootstrapping a Node Creating the Chef Server For this course, we're going to use the most recent stable release of chef server (which is 12.17.33). Our first step is going to be creating a CentOS 7 cloud server to be used as our Chef server. After that server is running, we need to get the download link for a Redhat 64bit system from the downloads page (here's the exact link used in this video). Let's use curl to copy this onto our server so that we can install it. Since we'll need to run virtually all of these commands using sudo we'll switch to root at the start: $ sudo su - [root] $ cd /tmp [root] $ curl -O https://packages.chef.io/files/stable/chef-server/12.17.33/el/7/chef-server-core-12.17.33-1.el7.x86_64.rpm [root] $ rpm -Uvh chef-server-core-12.17.33-1.el7.x86_64.rpm This might take a few moments, but this will install Cher Server and provide us with a few more utilities. Chef Server is a collection of many different services, and we'll need to configure those before we can do anything with the server. Chef Server utilizes Chef itself to configure its own services and we'll see this in the output from our next command: Note: This command can take quite awhile complete. [root] $ chef-server-ctl reconfigure Creating Our User and Organization Before we can move on, we need to set up a user and an organization to belong to. Organizations are the umbrella that we will register nodes under and associate cookbooks with as we move forward. Let's start by creating our user: [root] $ # chef-server-ctl user-create USER_NAME FIRST_NAME LAST_NAME EMAIL 'PASSWORD' --filename FILE_NAME [root] $ chef-server-ctl user-create keith Keith Thompson keith@linuxacademy.com 'p@ssw0rd' --filename /home/user/keith.pem This will create a username of keith and some extra metadata. An important thing to notice here is that the --filename output is an RSA key that we can use to interact with the Chef server from our workstation later on. Our next step will be creating an organization and setting ourselves as the first admin user. We can do this using a similar subcommand on the chef-server-ctl utility: [root] $ # chef-server-ctl org-create SHORT_ORG_NAME 'FULL_ORG_NAME' --association_user USER_NAME --filename FILE_NAME [root] $ chef-server-ctl org-create linuxacademy 'Linux Academy, Inc.' --association_user keith --filename linuxacademy-validator.pem This command is very similar to our user creation command, but it is important to note that there are some validation rules for the SHORT_ORG_NAME that we need to follow: Must start with a lowercase letter or number.Can only contain lowercase letters, digits, hyphens, and underscores.Must be between 1 and 255 characters long. The --association_user flag will take an existing user's username and associate it with the admin security group on the Chef server. Lastly, the --filename flag stores off the organization's validator pem. We won't be using this file during this course. Setting Up Our Workstation We'll continue utilizing CentOS 7 servers through this course, including our workstation. Once the server is running, we're ready to download the ChefDK rpm and install it. Since we're preparing for the exam, we'll be using the most recent release of the ChefDK that uses Chef 13 (which is 2.5.13 at this time): [workstation] $ cd /tmp [workstation] $ curl -O https://packages.chef.io/files/stable/chefdk/2.5.13/el/7/chefdk-2.5.13-1.el7.x86_64.rpm [workstation] $ sudo rpm -Uvh chefdk-2.5.13-1.el7.x86_64.rpm Note: chef-client version 14 has been released, but is not what the test is currently based around, so make sure you're using version 13.x. Most of the tooling around Chef is written in Ruby, and this can be a source of issues when working with Chef locally sometimes because we might already have a version of Ruby installed. Before we continue let's look at what version we might be using and see how to explicitly use the ChefDK version of these packages: [workstation] $ which ruby /usr/local/rvm/rubies/ruby-2.4.1/bin/ruby To ensure that we're using the proper version of Ruby we're going to add a line to our ~/.bash_profile to initialize the Chef environment. [workstation] $ echo 'eval "$(chef shell-init bash)"' >> ~/.bash_profile [workstation] $ source ~/.bash_profile Generating a chef-repo When developing with Chef we'll usually be working from within a "chef-repo" that holds onto our cookbooks and dependencies and is shared amongst our team(s). Let's create one now: [workstation] $ cd ~ [workstation] $ chef generate repo chef-repo ... [workstation] $ cd chef-repo [workstation] $ rm -rf cookbooks/example environments/example.json roles/example.json Utilizing the Chef generators makes it a lot easier for us to follow standard Chef development practices as we continue through this course. The main tool that we're going to use to connect to the Chef server is going to be the knife utility. A big difference between this repository and the one that we'll let the Chef server generate for us is that it's not pre-configured to allow us to use knife to connect to our Chef server. Let's manually create a knife configuration to communicate with the server: [workstation] $ knife configure WARNING: No knife configuration file found. See https://docs.chef.io/config_rb_knife.html for details. Please enter the chef server URL: [https://keiththomps5.mylabserver.com/organizations/myorg] https://keiththomps4.mylabserver.com/organizations/linuxacademy Please enter an existing username or clientname for the API: [user] keith ***** You must place your client key in: /home/user/.chef/keith.pem Before running commands with Knife ***** Configuration file written to Things to note are that we need to use the hostname from our Chef server (in this case it's keiththomps4.mylabserver.com) and also substitute in our organization's "short_name" (linuxacademy in this example). Next, we need to copy over the pem file that we created when we created our user. [workstation] $ scp user@keiththomps4.mylabserver.com:/home/user/keith.pem ~/.chef/ Before we can connect to the Chef Server we'll need to pull down the certificate as a trusted cert since it is self-signed. From there we can query the Chef Server for nodes: [workstation] $ knife ssl fetch WARNING: Certificates from keiththomps4.mylabserver.com will be fetched and placed in your trusted_cert directory (/home/user/.chef/trusted_certs). Knife has no means to verify these are the correct certificates. You should verify the authenticity of these certificates after downloading. Adding certificate for keiththomps4_mylabserver_com in /home/user/.chef/trusted_certs/keiththomps4_mylabserver_com.crt [workstation] $ knife node list The self-signed certificate for our Chef server is based on the hostname, so we'll use that in our knife configuration so that it will be correct even beyond a server restart with a new IP address. Running knife node list doesn't output anything because we don't have any nodes yet, but it didn't error so we know the connection worked. Bootstrapping Our First Node The last thing that we need to set up for this course is a test node that we can deploy our cookbooks to. For this, we'll need to spin up a third cloud server, reset the password, and then use knife to bootstrap it with Chef. After the password is reset, we can run the following from our workstation node. Note: Substitute the name of your cloud server for keiththomps3.mylabserver.com. [workstation] $ # knife bootstrap FQDN -N NODE_NAME -x USER -P PASSWORD --sudo [workstation] $ knife bootstrap keiththomps3.mylabserver.com -N web-node1 -x user -P 'p@ssw0rd' --sudo This process installs the chef-client and associated software packages onto the node and also utilizes the information from our workstation to communicate with the Chef Server and register the node as both a "node" and a "client" of the Chef server. Additionally, the chef-client is run, fetching the configuration from the Chef server, but there currently isn't anything there so the process only actually updates the Chef server with the information about the node. Ensuring Cloud Server Communication When starting & stopping cloud servers the public IP address changes, but the private ones do not. To remove DNS from the equation we're going to set up some entries in the /etc/hosts file on our workstation and web-node1 so that they both know how to communicate with the Chef Server and also so that the workstation knows how to reach web-node1 using the private network. The entries will look something like this for the workstation: /etc/hosts 172.31.42.111 keiththomps1.mylabserver.com 172.31.43.241 keiththomps3.mylabserver.com 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 And like this for web-node1: /etc/hosts 172.31.42.111 keiththomps1.mylabserver.com 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 Now we're ready to follow along with the rest of this course.

Working Around A Bug In Knife

00:03:51

Lesson Description:

No software is perfect and occasionally we need to work around bugs in the tools that we use. In this lesson, we'll show how to work around a bug with certain Knife commands when you use a Chef Repo that doesn't have a knife.rb file. Note: If the issue down below has been closed then your version of Knife might not have this issue. Documentation For This Video Related Github Issue Working Around knife upload Issues The bug that we're running into is very specific and will only affect us if we have done ALL of the following: Generated our chef-repo using chef generate repo.Configured Knife using knife configure.Never manually created a knife.rb file within the chef-repo. Essentially, the issue is that when certain Knife commands run the cookbook_path Knife configuration value needs to be set properly and a change to how knife configure worked in December 2017 leads to this not being configured when it is documented to be. To see if you have this issue, attempt to upload a cookbook to the Chef Server using knife upload. We're going to get around this by creating a file at ~/chef-repo/.chef/knife.rb. [workstation] chef-repo $ mkdir .chef [workstation] chef-repo $ echo 'cookbook_path ["#{File.dirname(__FILE__)}/../cookbooks"]' > .chef/knife.rb Now if we're in our chef-repo then Knife will have a properly set cookbook_path.

Extending Ohai

Basic Ohai Plugin Authoring

What is Ohai?

00:02:45

Lesson Description:

As you've used Chef you've probably noticed that there is a ton of information about each node stored in the Chef Server, but how does that information get there? The answer is Ohai and in this lesson, we'll take a look at what exactly Ohai is. Documentation For This Video Chef's Ohai documentationBasic Chef Fluency Badge coursePDF of video's slideshow Collecting All of the Data If you've gone through the Basic Chef Fluency Badge course, then you have been exposed to node attributes and the various attribute levels. These levels can be pretty complicated, but they're a lot easier when it comes to information collected by Ohai. All of the attributes collected by Ohai fall under the level of "automatic," and cannot be overwritten. The data collection that Ohai performs is done prior to any resources being executed. But occasionally, you'll want to reload this information later on in a chef-client run, once you've made a change that should be reflected in an automatic attribute.

The Ohai DSL

00:05:40

Lesson Description:

Before we dig too far into writing and deploying custom Ohai plugins, we're going to take the time to read an existing one, to get an understanding of the Ohai DSL (Domain Specific Language). Documentation For This Video Chef's Ohai documentationWriting Custom Ohai Plugins Plugins and Where to Find Them One of the major requirements for the Extending Chef Badge exam is the ability to write custom Ohai plugins. Since plugins are used to provide the built-in information that is collected, it is definitely worth knowing where those plugins are stored on a node. By default, we'll be able to find them in the lib/ohai/plugins directory for the gem. Here are a few commands to run to on a node to see all of the available plugins for the current version of Ohai. First get the version: [root] $ ohai --version Ohai: 14.3.1 Then use that in the following find command: [root] $ find / -path "*embedded*14.3.1/lib/ohai/plugins/*" ... Note: That list should be pretty long The os Plugin In the previous lesson, we briefly showed the built-in os plugin for Ohai. Now let's have a look at the os.rb file: Ohai.plugin(:OS) do require "ohai/mixin/os" provides "os", "os_version" depends "kernel" collect_data do os collect_os os_version kernel[:release] end end Let's figure out what parts of it are DSL, and what are "just Ruby." Ohai.plugin(:OS) do - Ohai DSL: This is where the plugin is given a name.require "ohai/mixin/os" - Pure Ruby: This loads a different piece of code from within the ohai gem into the plugin's context.provides "os", "os_version" - Ohai DSL: This line specifies the top level attributes that this plugin will provide. From here we know that this is where node['os'] comes from.depends "kernel" - Ohai DSL: Here, it is stating that this plugin requires information that is provided by the kernel Ohai plugin.collect_data do - Ohai DSL: This is the primary method we use when writing an Ohai plugin. Within this block, we need to specify the contents of the attributes we will be providing (in this case os and os_version).os collect_os - Ohai DSL: The collect_os method is provided by the ohai/mixin/os that was loaded earlier. The os method is created from the provides... line, and is how we specify the data to return for that attribute.os_version kernel[:release] - Ohai DSL: The os_version method works just like the os method from the previous line. The kernel[:release] portion is available because of the depends "kernel" line earlier in the file and it is returning the :release data that was collected by the kernel plugin. When it comes to working with Ohai, most of what we'll be working with is going to be part of the Ohai DSL. This can be a little confusing at times. Once we learn how to distinguish what is what, it will become very easy for us to write custom Ohai plugins.

The Ohai Cookbook

Creating an Ohai Plugin

00:06:19

Lesson Description:

In this lesson, we're going to start extending Chef by creating our first custom Ohai plugin. We'll start small and then expand as we continue on. Documentation For This Video Chef's Ohai documentationWriting Custom Ohai PluginsThe ohai community cookbook Preparing Our Cookbook Similar to the approach that we took in the Basic Chef Fluency course we're going to use NGINX as our first sample cookbook and Ohai plugin. Let's create the cookbook and add the resources necessary to install the NGINX package. We're going to prefix the cookbooks from this course with ec_ (for "Extending Chef"). [workstation] chef-repo $ chef generate cookbook cookbooks/ec_nginx ... [workstation] chef-repo $ cd cookbooks/ec_nginxWe'll quickly set up the cookbook to install the epel-release (for Red Hat systems) and nginx packages. ~/chef-repo/cookbooks/ec_nginx/recipes/default.rb if node['platform_family'] == 'rhel' package 'epel-release' end package 'nginx' service 'nginx' do action [:enable, :start] endNow we're ready to create our plugin to add some NGINX specific information to the node's attributes. Writing The nginx Ohai Plugin Ohai plugins are Ruby files and they don't have to be part of cookbooks to be deployed (we could manually put the plugin file on the server). In our case, we're packaging this up with our cookbook for deployment and we'll need to put the file in our files directory. [workstation] ec_nginx $ chef generate file nginx.rb ...Now let's go through and register our plugin to provide the nginx/version attribute. ~/chef-repo/cookbooks/ec_nginx/files/default/nginx.rb Ohai.plugin(:Nginx) do provides 'nginx/version' collect_data :default do nginx(Mash.new) nginx[:version] = shell_out('nginx --version').stdout end endThis is obviously not the final product, but we want to see if it works. We're specifying that the Plugin name is :Nginx and that it provides the nginx/version attribute. Ohai will create an nginx method for us to set up the Hash or Mash to return as the data. For now, we're then setting the value of :version to the entire value of STDOUT from the shell_out('nginx --version']) command. Note: A Mash is a data type that works like a Hash except that you can use symbols and strings as keys interchangeably. In the next lesson, we'll learn how to test this locally using Kitchen and also how to deploy it to our web-node1 node.

Testing & Deploying an Ohai Plugin

00:12:57

Lesson Description:

Now that we've written a simple Ohai plugin we need to deploy it to see that it works, but we're going to start by writing some local automated tests. Documentation For This Video KitchenInstalling Docker on CentOSThe ohai community cookbook Setting Up Kitchen + Docker For our testing we're going to use Kitchen, InSpec, and Docker via kitchen-docker. This will allow us to spin up containers quickly to run our cookbook recipes and test our plugins, handlers, etc. as we go through this course. We already have both kitchen and inspec installed as part of the ChefDK, so we need to install Docker and then kitchen-docker now. Note: I'm assuming you're following along on a CentOS 7 machine. If not follow the Docker installation directions for your operating system. [workstation] $ sudo yum install -y yum-utils device-mapper-persistent-data lvm2 ... [workstation] $ sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo ... [workstation] $ sudo yum install -y docker-ce ... [workstation] $ sudo systemctl start docker; sudo systemctl enable dockerNow we have Docker installed, running, and enabled to start on boot. I don't want to have to type sudo every time I use docker so I'm also going to add my user to the docker group. [workstation] $ sudo usermod -a -G docker $(whoami)We'll need to log out and log back in for the group to load. After logging back in we should be able to use Docker without using sudo. Next, we'll install the kitchen-docker gem using chef gem install kitchen-docker: [workstation] $ chef gem install kitchen-docker ...Now we're ready to use Kitchen and Docker in the integration tests for our cookbooks. We next need to configure the .kitchen.yml file to use the docker driver. ~/chef-repo/cookbooks/ec_nginx/.kitchen.yml --- driver: name: docker use_sudo: false provisioner: name: chef_zero # You may wish to disable always updating cookbooks in CI or other testing environments. # For example: # always_update_cookbooks: <%= !ENV['CI'] %> always_update_cookbooks: true product_name: chef product_version: 13.10.4 verifier: name: inspec platforms: - name: centos-7 driver_config: run_command: /usr/sbin/init privileged: true provision_command: - systemctl enable sshd.service suites: - name: default run_list: - recipe[ec_nginx::default] verifier: inspec_tests: - test/integration/default attributes:Note: We're using Chef 13.10.4 because it's the newest Chef 13, which is covered by the exam. There's a little bit more going on than usual because we need to make sure that we can start services within the container. Before continuing, run kitchen test from within the cookbook to ensure that it converges successfully. Testing an Ohai Plugin To test an Ohai plugin using Kitchen and InSpec we'll need to do the following: Create a recipe that will install our plugin on the node.Add a new test under the test directory. We'll start with the test first. We're going to call the recipe ec_nginx::ohai so let's create the test at test/integration/default/ohai_test.rb ~/chef-repo/cookbooks/ec_nginx/test/integration/default/ohai_test.rb describe command("ohai -d /tmp/kitchen/ohai/plugins nginx") do its(:stdout) { should include('nginx version: nginx/1.12.2') } endNow we need to generate the recipe: [workstation] ec_nginx $ chef generate recipe ohaiLet's also add it to the testing run_list: ~/chef-repo/cookbooks/ec_nginx/.kitchen.yml # Previous lines omitted suites: - name: default run_list: - recipe[ec_nginx::default] - recipe[ec_nginx::ohai] verifier: inspec_tests: - test/integration/default attributes:Finally, let's run the test to make sure that it fails [workstation] ec_nginx $ kitchen test ... Profile: tests from {:path=>"/home/user/chef-repo/cookbooks/ec_nginx/test/integration/default"} (tests from {:path=>".home.user.chef-repo.cookbooks.ec_nginx.test.integration.default"}) Version: (not specified) Target: ssh://kitchen@localhost:32776 User root ? Port 80 ? Command ohai -d /tmp/kitchen/ohai/plugins nginx ? stdout should include "nginx version: nginx/1.12.2" expected "" to include "nginx version: nginx/1.12.2" Test Summary: 0 successful, 1 failure, 2 skippedNow that we have a failing test we're ready to make it pass. Using the ohai Community Cookbook Deploying Ohai plugins requires that we place our Ruby file in the proper directory on the node which sounds simple, but we want everyone using the same directory so this is where the ohai cookbook comes in. Let's add this dependency and utilize it in our ohai recipe. ~/chef-repo/cookbooks/ec_nginx/metadata.rb name 'ec_nginx' maintainer 'Keith Thompson' maintainer_email 'keith@linuxacademy.com' license 'All Rights Reserved' description 'Installs/Configures NGINX' long_description 'Installs/Configures NGINX' version '0.1.0' chef_version '>= 12.14' if respond_to?(:chef_version) depends 'ohai', '~> 5.2.3'~/chef-repo/cookbooks/ec_nginx/recipes/ohai.rb ohai_plugin 'nginx'The ohai_plugin resource allows us to specify the name of our plugin file and then the ohai cookbook will handle the rest. Now we need to download this using berks install and we'll be ready to run our test again. [workstation] ec_nginx $ berks install ... [workstation] ec_nginx $ kitchen test ... Profile: tests from {:path=>"/home/user/chef-repo/cookbooks/ec_nginx/test/integration/default"} (tests from {:path=>".home.user.chef-repo.cookbooks.ec_nginx.test.integration.default"}) Version: (not specified) Target: ssh://kitchen@localhost:32777 User root ? Port 80 ? Command ohai -d /tmp/kitchen/ohai/plugins nginx ? stdout should include "nginx version: nginx/1.12.2" expected "{n "version": ""n}n" to include "nginx version: nginx/1.12.2" Diff: @@ -1,2 +1,4 @@ -nginx version: nginx/1.12.2 +{ + "version": "" +} Test Summary: 0 successful, 1 failure, 2 skippedIt's not passing, but we know that our recipe would successfully add the plugin for use by Ohai. Now is a good time for us to investigate how we might debug an issue like this (in the next lesson).

Hands-on Labs are real live environments that put you in a real scenario to practice what you have learned without any other extra charge or account to manage.

01:00:00

Troubleshooting Ohai Plugins

Debugging With Chef-Shell

00:09:04

Lesson Description:

The more custom code that we write the more likely we are to run into bugs. More code equals more potential bugs. In this lesson, we'll take a look at a few of the tools provided by Chef and Ruby to allow for rapid debugging. Documentation For This Video The chef-shell documentationChef's debugging documentationThe ohai Github Repository Utilizing chef-shell to Debug an Ohai Plugin Currently, our nginx Ohai plugin is providing the nginx/version attribute, but it is not giving us the information that we are expecting. This points us towards thinking that there's something wrong with the shell_out(..) line in our plugin. To see what this line is doing we're going to log into our testing container using kitchen login default-centos-7: [workstation] ec_nginx $ kitchen login default-centos-7 Last login: Wed Aug 15 14:29:01 2018 from gateway [kitchen@205c1cd94218 ~]$ From within the container, we can use chef-shell to load into a Chef like environment that acts like a REPL (Read Evaluate Print Loop): [kitchen@205c1cd94218 ~]$ chef-shell loading configuration: none (standalone session) Session type: standalone Loading.[2018-08-15T14:33:16+00:00] WARN: Plugin Network: unable to detect ipaddress ..done. This is the chef-shell. Chef Version: 13.10.4 https://www.chef.io/ https://docs.chef.io/ run `help' for help, `exit' or ^D to quit. Ohai2u kitchen@205c1cd94218! chef (13.10.4)> Now we can execute Ruby/Chef code and see what is returned. We're interested in what shell_out('nginx --version') returns to us. chef (13.10.4)> shell_out('nginx --version') NoMethodError: undefined method `shell_out' for main:Object from (irb):1 from /opt/chef/embedded/lib/ruby/gems/2.4.0/gems/chef-13.10.4/lib/chef/shell.rb:82:in `block in start' from /opt/chef/embedded/lib/ruby/gems/2.4.0/gems/chef-13.10.4/lib/chef/shell.rb:81:in `catch' from /opt/chef/embedded/lib/ruby/gems/2.4.0/gems/chef-13.10.4/lib/chef/shell.rb:81:in `start' from /opt/chef/embedded/lib/ruby/gems/2.4.0/gems/chef-13.10.4/bin/chef-shell:36:in `<top (required)>' from /usr/bin/chef-shell:75:in `load' from /usr/bin/chef-shell:75:in `<main>' That's a problem, the shell_out method isn't available to us (just yet). We know that we do have access to this method within an Ohai plugin, so it's safe to assume that Ohai itself provides this to us. Doing a search in the chef/ohai Github repository leads us to see that it is defined in the ohai/mixin/command module. The implementation shows that it's creating a Mixlib::ShellOut object and then running run_command. We can do this manually to see what happens when we run the nginx --version command. chef (13.10.4)> command = Mixlib::ShellOut.new('nginx --version') => <Mixlib::ShellOut#27122420: command: 'nginx --version' process_status: nil stdout: '' stderr: '' child_pid: nil environment: {} timeout: 600 user: group: working_dir: > chef (13.10.4)> command.run_command => <Mixlib::ShellOut#27122420: command: 'nginx --version' process_status: #<Process::Status: pid 2515 exit 1> stdout: '' stderr: 'nginx: invalid option: "-"' child_pid: 2515 environment: {} timeout: 600 user: group: working_dir: > chef (13.10.4)> The object's information shows that stdout is empty, but stderr has a message. It looks like --version isn't the right flag. Let's try with the -v flag instead. chef (13.10.4)> command = Mixlib::ShellOut.new('nginx -v') => <Mixlib::ShellOut#26502100: command: 'nginx -v' process_status: nil stdout: '' stderr: '' child_pid: nil environment: {} timeout: 600 user: group: working_dir: > chef (13.10.4)> command.run_command => <Mixlib::ShellOut#26502100: command: 'nginx -v' process_status: #<Process::Status: pid 2530 exit 0> stdout: '' stderr: 'nginx version: nginx/1.12.2' child_pid: 2530 environment: {} timeout: 600 user: group: working_dir: > It turns out that the -v flag is correct, but the string we're expecting it in the stderr attribute on the object instead of stdout. Now we know how to change our plugin: ~/chef-repo/ec_nginx/files/default/nginx.rb Ohai.plugin(:Nginx) do provides 'nginx/version' collect_data :default do nginx(Mash.new) nginx[:version] = shell_out('nginx -v').stderr end end One of the perks of using the chef-shell is that it converges ohai before placing us at the prompt so we have access to the node just like we would in our recipes. This can be handy when debugging recipes and resources. Let's run kitchen test to make sure that our plugin is reporting the data that we expect. [workstation] ec_nginx $ kitchen test ... Profile: tests from {:path=>"/home/user/chef-repo/cookbooks/ec_nginx/test/integration/default"} (tests from {:path=>".home.user.chef-repo.cookbooks.ec_nginx.test.integration.default"}) Version: (not specified) Target: ssh://kitchen@localhost:32769 User root ? Port 80 ? Command ohai -d /tmp/kitchen/ohai/plugins nginx ? stdout should include "nginx version: nginx/1.12.2" Test Summary: 1 successful, 0 failures, 2 skipped Finished verifying <default-centos-7> (0m0.79s). -----> Destroying <default-centos-7>... Experimenting with Ruby Using IRB We didn't actually need the node attributes to experiment with the piece of code that we thought was failing. We could have used the REPL that comes with Ruby called irb. It would have loaded us into a Ruby environment that didn't have pieces of Chef preloaded. I encourage you to use irb to experiment with Ruby's own features as you're becoming more and more comfortable extending Chef. Don't Be Afraid to Read Code To be truly comfortable extending Chef we need to be willing to read the Chef's own code, community cookbooks, and always be learning. A lot can be learned by reading other people's code.

Loading and Reloading Ohai

00:07:47

Lesson Description:

We've tested our NGINX Ohai plugin locally and now we're ready to deploy to an actual server. Our confidence that it will be deployed to the machine should be pretty high, but that doesn't mean that it will work perfectly the first time. In this lesson, we'll look at how the execution of Ohai plugins can be changed in recipes. Documentation For This Video The Ohai DocumentationThe Ohai Custom Plugin Documentation Deploying To Our Node Running tests to verify that Ohai is working is good, but now we want to deploy this to our node. We need to do the following to make sure that it works: Upload cookbooks to the Chef Server using berks uploadSet the run-list on web-node1 to recipe[ec_nginx::default],recipe[ec_nginx::ohai]Run chef-client on web-node1Verify the information was stored to the node using knife node show web-node1 Once we do all of this we'll see the following: [workstation] ec_nginx $ berks upload ... [workstation] ec_nginx $ knife node run_list set web-node1 'recipe[ec_nginx::default],recipe[ec_nginx::ohai]' web-node1: run_list: recipe[ec_nginx::default] recipe[ec_nginx::ohai] [workstation] ec_nginx $ knife ssh 'name:web-node1' 'sudo chef-client' ... [workstation] ec_nginx $ knife node show web-node1 -F json -a nginx { "web-node1": { "nginx": { } } } It's close, but why was the version stored? The ohai_plugin resource runs at compile time (so before the other resource run) which means that we installed the plugin and it ran before NGINX was installed. If we run chef-client again and check our data again we will see that the value is set: [workstation] ec_nginx $ knife ssh 'name:web-node1' 'sudo chef-client' ... keiththomps3.mylabserver.com Chef Client finished, 0/12 resources updated in 06 seconds [workstation] ec_nginx $ knife node show web-node1 -F json -a nginx { "web-node1": { "nginx": { "version": "nginx version: nginx/1.12.2n" } } } This happens because the automatic attributes are calculated at the beginning of each chef-client run and then reported to the Chef Server at the end of the run (even if no resources run). How can we adjust our recipe to ensure that the data is collected the first time we run our recipes on a node? Reloading Ohai During a Chef Client Run To interact with Ohai from within a recipe we have the [ohai][4] resource that is provided by Chef instead of an external cookbook. This resource has one useful action, the :reload action. Using this resource and the notification system we can ensure that our Ohai plugin regathers data if there is a change to the package[nginx] resource. ~/chef-repo/ec_nginx/recipes/ohai.rb ohai_plugin 'nginx' ohai 'reload_nginx' do plugin 'nginx' action :nothing subscribes :reload, 'package[nginx]' end To test this out we will need to remove NGINX from web-node1, upload the updated cookbook, and run chef-client again. [workstation] ec_nginx $ knife ssh 'name:web-node1' 'sudo yum remove -y nginx' ... [workstation] ec_nginx $ berks upload --force ... [workstation] ec_nginx $ knife ssh 'name:web-node1' 'sudo chef-client' ... keiththomps3.mylabserver.com Compiling Cookbooks... keiththomps3.mylabserver.com Recipe: ec_nginx::ohai keiththomps3.mylabserver.com * ohai_plugin[nginx] action create keiththomps3.mylabserver.com * directory[/etc/chef/ohai/plugins] action create (skipped due to not_if) keiththomps3.mylabserver.com * cookbook_file[/etc/chef/ohai/plugins/nginx.rb] action create (up to date) keiththomps3.mylabserver.com * ohai[nginx] action nothing (skipped due to action :nothing) keiththomps3.mylabserver.com (up to date) keiththomps3.mylabserver.com Converging 5 resources keiththomps3.mylabserver.com Recipe: ec_nginx::default keiththomps3.mylabserver.com * yum_package[epel-release] action install (up to date) keiththomps3.mylabserver.com * yum_package[nginx] action install keiththomps3.mylabserver.com - install version 1.12.2-2.el7 of package nginx keiththomps3.mylabserver.com * service[nginx] action enable keiththomps3.mylabserver.com - enable service service[nginx] keiththomps3.mylabserver.com * service[nginx] action start keiththomps3.mylabserver.com - start service service[nginx] keiththomps3.mylabserver.com Recipe: ec_nginx::ohai keiththomps3.mylabserver.com * ohai_plugin[nginx] action create keiththomps3.mylabserver.com * directory[/etc/chef/ohai/plugins] action create (skipped due to not_if) keiththomps3.mylabserver.com * cookbook_file[/etc/chef/ohai/plugins/nginx.rb] action create (up to date) keiththomps3.mylabserver.com * ohai[nginx] action nothing (skipped due to action :nothing) keiththomps3.mylabserver.com (up to date) keiththomps3.mylabserver.com * ohai[reload_nginx] action nothing (skipped due to action :nothing) keiththomps3.mylabserver.com * ohai[reload_nginx] action reload keiththomps3.mylabserver.com - re-run ohai and merge results into node attributes keiththomps3.mylabserver.com keiththomps3.mylabserver.com Running handlers: keiththomps3.mylabserver.com Running handlers complete keiththomps3.mylabserver.com Chef Client finished, 4/14 resources updated in 10 seconds Now if we look one more time we should see that the nginx/version attribute is still set properly. [workstation] ec_nginx $ knife node show web-node1 -F json -a nginx { "web-node1": { "nginx": { "version": "nginx version: nginx/1.12.2n" } } }

Ohai Hints

00:10:28

Lesson Description:

Every now and then we need Ohai to be aware of information that it can't gather itself reliably, and for that, we have "hints". Documentation For This Video Chef Ohai documentationChef Ohai Hints documentationHints used within Ohai Creating and Accessing a Hint Hints are JSON files that live on our nodes and provide optional information that Ohai can't easily figure out by itself. Ohai is great at accessing information that is one the node itself through various commands or viewing what is on the file system, but occasionally it needs to be influenced by an outside source or would need to make a network call to get the information (for instance EC2 metadata). Hints are created by putting JSON files within the hints path (the default is /etc/chef/ohai/hints/). To test this out we're going to create a hint file on our workstation and access it from the chef-shell. [workstation] chef-repo $ sudo mkdir -p /etc/chef/ohai/hints [workstation] chef-repo $ echo '{"members": ["wolverine", "cyclops", "beast", "storm"]}' | sudo tee /etc/chef/ohai/hints/xmen.json {"members": ["wolverine", "cyclops", "beast", "storm"]} Now we can open up the chef-shell to see how we would use this: chef (13.10.4)> Ohai::Hints.hint?('xmen') => {"members"=>["wolverine", "cyclops", "beast", "storm"]} From within our plugins, we don't need to use the full module name because there is a method provided to the plugin called hints?, but underneath it would call this exact line. Since the name of the file is xmen.json the hint? method will store the contents as a hash underneath the key of xmen. When are Hints Useful? The best way to see how hints can be useful is to look within Ohai itself. If we search for [where hints are used within the standard library of Ohai plugins][4] we can see that it's almost always to access metadata provided by a cloud provider.

Enabling & Disabling OHAI Plugins

Ohai 'client.rb' Settings

00:12:00

Lesson Description:

Ohai isn't only customizable by adding custom plugins, we can configure it to meet our specific needs as a part of the Chef Client process. In this lesson, we'll take a look at how we can configure Ohai using a node's client.rb. Documentation For This Video Chef Ohai DocumentationOhai client.rb SettingsThe chef-client cookbook Ohai Settings Ohai is an integral part of the chef-client process and because of that it is configurable in the same place as the rest of the Chef Client via the client.rb file that every node has at /etc/chef/client.rb. There are quite a few different Ohai settings available to use (which we can find in the documentation), but one that might be particularly important is the ability to disable plugins. Having more Ohai plugins enabled means that more modules need to be loaded and more data collected, even if we don't care about the data that the module provides. Disabling Ohai Plugins To demonstrate how adjusting Ohai settings can be done, we're going to utilize the chef-client cookbook. This cookbook is used to allow Chef to manage it's own configuring using node attributes to dynamically generate the client.rb, and you'll also use this cookbook to configure chef-client to run on an interval to periodically check in with the Chef Server. For now, let's create a cookbook called ec_base where we'll put our general node configuration and wrap the chef-client cookbook. [workstation] chef-repo $ chef generate cookbook cookbooks/ec_base ... [workstation] chef-repo $ cd cookbooks/ec_base Next, we'll add a dependency on the chef-client cookbook to our metadata.rb. ~/chef-repo/cookbooks/ec_base/metadata.rb # Existing content omitted depends 'chef-client', '~> 11.0.0' We need to pull down the cookbook using Berkshelf. [workstation] ec_base $ berks install Resolving cookbook dependencies... Fetching 'ec_base' from source at . Fetching cookbook index from https://supermarket.chef.io... Installing chef-client (11.0.0) Installing cron (6.2.0) Using ec_base (0.1.0) from source at . Installing logrotate (2.2.0) Finally, before we push this cookbook to our Chef Server we're going to create an attributes file called chef-client.rb [workstation] ec_base $ chef generate attribute chef-client Recipe: code_generator::attribute * directory[/home/user/chef-repo/cookbooks/ec_base/attributes] action create - create new directory /home/user/chef-repo/cookbooks/ec_base/attributes - restore selinux security context * template[/home/user/chef-repo/cookbooks/ec_base/attributes/chef-client.rb] action create - create new file /home/user/chef-repo/cookbooks/ec_base/attributes/chef-client.rb - update content in file /home/user/chef-repo/cookbooks/ec_base/attributes/chef-client.rb from none to e3b0c4 (diff output suppressed by config) - restore selinux security context This attributes files gives us a way to easily configure the chef-client settings that we'll put on all of our nodes, and from here we can disable a few Ohai plugins that we don't care about. Let's take a look at the plugins available to find one to ignore: [workstation] ec_base $ sudo find / -path '*lib/ohai/plugins/*' ... We're not going to use Perl on our servers so let's prevent that plugin from running. Beforehand we should see what it provides so we can do a comparison. [workstation] ec_base $ cat $(sudo find / -path '*lib/ohai/plugins/perl.rb') Ohai.plugin(:Perl) do provides "languages/perl" depends "languages" collect_data do begin so = shell_out("perl -V:version -V:archname") # Sample output: # version='5.18.2'; # archname='darwin-thread-multi-2level'; if so.exitstatus == 0 perl = Mash.new so.stdout.split(/r?n/).each do |line| case line when /^version='(.+)';$/ perl[:version] = $1 when /^archname='(.+)';$/ perl[:archname] = $1 end end languages[:perl] = perl unless perl.empty? end rescue Ohai::Exceptions::Exec Ohai::Log.debug('Plugin Perl: Could not shell_out "perl -V:version -V:archname". Skipping plugin') end end end [workstation] ec_base $ knife node show web-node1 -a languages.perl web-node1: languages.perl: archname: x86_64-linux-thread-multi version: 5.16.3 Now we can disable this plugin by adding it to the disabled_plugins list using its registered name: ~/chef-repo/cookbooks/ec_base/attributes/chef-client.rb node.default['ohai']['disabled_plugins'] = [ :Perl ] We do need to include the chef-client::config recipe in our ec_base/recipes/default.rb before we can verify that this worked. ~/chef-repo/cookbooks/ec_base/recipes/default.rb include_recipe 'chef-client::config' ohai 'reload_after_config_change' do action :nothing subscribes :reload, 'template[/etc/chef/client.rb]', :immediately end Note: We're reloading Ohai after the client.rb changes so that we don't have stale data in the node object after the initial run with this recipe. Finally, we'll deploy this, run chef-client, and ensure that we're no longer gathering the languages.perl attribute: [workstation] ec_base $ berks upload ... [workstation] ec_base $ knife node run_list add web-node1 ec_base --before 'recipe[ec_nginx::default]' web-node1: run_list: recipe[ec_base] recipe[ec_nginx::default] recipe[ec_nginx::ohai] [workstation] ec_base $ knife ssh 'name:web-node1' 'sudo chef-client' ... Let's see if it stopped tracking the information on Perl: [workstation] ec_base $ knife node show web-node1 -a languages.perl web-node1: languages.perl:

Blacklisting vs Whitelisting Attributes

00:11:27

Lesson Description:

By disabling Ohai plugins will prevent the automatic attributes from being collected by the plugin during the Chef run, but what if we simply don't want to save the values to the Chef Server? To facilitate that we can utilize the attribute "blacklist" or "whitelist". Documentation For This Video Chef blacklist attributesChef whitelist attributes Blacklisting and Whitelisting Attributes When it comes to blacklisting or whitelisting attributes we'll be heading back into the client.rb on our nodes to adjust settings. This time around we won't be working off of the ohai section, but rather calling methods directly in the file based on the attributes types. The methods are as follows: automatic_attribute_whitelistautomatic_attribute_blacklistdefault_attribute_whitelistdefault_attribute_blacklistnormal_attribute_whitelistnormal_attribute_blacklistoverride_attribute_whitelistoverride_attribute_blacklist Each of these methods takes an array of attributes names (or an empty array) or nil as an argument. In all cases, if nil or [] is given then all of the attributes of that type are saved as they would be normally. Blacklisting the languages Attributes For our example, let's stop storing the languages attributes using the automatic_attribute_blacklist method by editing our ec_base/attributes/chef-client.rb file. ~/chef-repo/cookbooks/ec_base/attributes/chef-client.rb node.default['ohai']['disabled_plugins'] = [ :Perl, ] node.default['chef_client']['config']['automatic_attribute_blacklist'] = [ 'languages' ] Before we upload this update and run chef-client on our node we should see what we get before we change the client.rb settings: [workstation] ec_base $ knife node show web-node1 -a languages web-node1: languages: c: glibc: description: GNU C Library (GNU libc) stable release version 2.17, by Roland McGrath et al. version: 2.17 lua: version: 5.1.4 python: builddate: Jul 13 2018, 13:06:57 version: 2.7.5 ruby: bin_dir: /usr/local/rvm/rubies/ruby-2.4.1/bin gem_bin: /usr/local/rvm/rubies/ruby-2.4.1/bin/gem gems_dir: /usr/local/rvm/rubies/ruby-2.4.1/lib/ruby/gems/2.4.0 host: x86_64-pc-linux-gnu host_cpu: x86_64 host_os: linux-gnu host_vendor: pc platform: x86_64-linux release_date: 2017-03-22 ruby_bin: /usr/local/rvm/rubies/ruby-2.4.1/bin/ruby target: x86_64-pc-linux-gnu target_cpu: x86_64 target_os: linux target_vendor: pc version: 2.4.1 Let's upload this change and run chef-client: [workstation] ec_base $ berks upload --force ... [workstation] ec_base $ knife ssh 'name:web-node1' 'sudo chef-client' ... keiththomps3.mylabserver.com * template[/etc/chef/client.rb] action create keiththomps3.mylabserver.com - update content in file /etc/chef/client.rb from e27978 to 8e5b6c keiththomps3.mylabserver.com --- /etc/chef/client.rb 2018-08-17 15:00:53.141373117 +0000 keiththomps3.mylabserver.com +++ /etc/chef/.chef-client20180817-1757-9wrgy2.rb 2018-08-17 17:56:03.998108345 +0000 keiththomps3.mylabserver.com @@ -1,3 +1,4 @@ keiththomps3.mylabserver.com +automatic_attribute_blacklist ["languages"] keiththomps3.mylabserver.com chef_server_url "https://keiththomps1.mylabserver.com/organizations/linuxacademy" keiththomps3.mylabserver.com client_fork true keiththomps3.mylabserver.com log_location "/var/log/chef/client.log" keiththomps3.mylabserver.com - restore selinux security context ... Note: we're using the --force flag here because we're going to rapidly be making modifications and don't want to update our version while we're experimenting. The chef-client cookbook translated our node.default['chef_client']['config']['automatic_attribute_blacklist'] = ['languages'] into this method call: automatic_attribute_blacklist ["languages"] Now let's see what is stored on the node: [workstation] ec_base $ knife node show web-node1 -a languages web-node1: languages: Caution: Whitelisting Attributes DON'T DO THIS The alternative to blacklisting would be whitelisting, but this is one that we should use with caution because it will only store the attributes that we specify and ignore the others (of the same type). I'm going to see what happens when we change from using blacklist to whitelist in our attributes file: ~/chef-repo/cookbooks/ec_base/attributes/chef-client.rb node.default['ohai']['disabled_plugins'] = [ :Perl, ] node.default['chef_client']['config']['automatic_attribute_whitelist'] = [ 'nginx/version' ] Now I'll deploy and run chef-client: [workstation] ec_base $ berks upload --force ... [workstation] ec_base $ knife ssh 'name:web-node1' 'sudo chef-client' ... Finally, we'll look at all of the attributes to see what remains (using the -a automatic flag to ignore any of the attributes that aren't automatic): [workstation] ec_base $ knife node show web-node1 -a automatic web-node1: automatic: nginx: version: nginx version: nginx/1.12.2 This is significantly less information, but the problem is that now we don't know the ipaddress or hostname of our node. [workstation] ec_base $ knife node show web-node1 -a ipaddress web-node1: ipaddress: Because of this, I now can't use knife ssh to connect to my node. I'll have to manually ssh to the node's IP address and rerun chef-client twice after changing the cookbook back to blacklisting. I would strongly caution against using a whitelist on automatic attributes and only use whitelisting at all if you have an exceptionally good reason. If you did follow along, please change the configuration back to blacklisting the languages key and use the following command with your node's IP address to get the data back into your Chef Server. You will need to run this twice: [workstation] $ ec_base $ ssh -t user@[YOUR_IP] 'sudo chef-client' ...

Hands-on Labs are real live environments that put you in a real scenario to practice what you have learned without any other extra charge or account to manage.

01:00:00

Custom Resources

Custom Resource DSL

What are Custom Resources?

00:04:43

Lesson Description:

Up to this point, we've used one resource that didn't come packaged with Chef itself. The ohai_plugin is an example of a custom resource, and in this lesson we'll look at when we might want to create our own custom resources. Documentation For This Video Chef Custom Resources DocumentationPDF of video's slideshow

Creating Custom Resources - Part 1

00:12:40

Lesson Description:

Now that we've seen our first custom resource we're ready to dig in and write our own. In this lesson, we'll take a look at what it would take to create our own custom resource to easily create NGINX virtual hosts. Documentation For This Video Chef Custom Resources DocumentationChef Custom Resource DSL The nginx_server Resource When it comes to setting up a new NGINX server (aka virtual host) there are a few things that we need to do: Create the configuration file for the server in /etc/nginx/conf.d/.Specify where static assets will come from. If we were to write a single recipe to do this we might use the following resources: directory - Ensure /etc/nginx/conf.d/ exists and that directory for static assets can be hosted.template - Create a dynamical server configuration file. Writing Some Tests Before we get started we're going to create a test recipe that maps out how we would like it to be used. We need to set up some directories for our test fixture recipes: [workstation] ec_nginx $ mkdir -p test/fixtures/cookbooks/nginx_test/recipes [workstation] ec_nginx $ touch test/fixtures/cookbooks/nginx_test/recipes/default.rb [workstation] ec_nginx $ touch test/fixtures/cookbooks/nginx_test/metadata.rb [workstation] ec_nginx $ mkdir spec/unit/resources/ Our metadata.rb is going to be pretty basic: ~/chef-repo/cookbooks/ec_nginx/test/fixtures/cookbooks/nginx_test/metadata.rb name 'nginx_test' maintainer 'John Doe' maintainer_email 'johndoe@example.com' license 'Apache-2.0' description 'Tests the ec_nginx cookbook resources' version '1.0.0' depends 'ec_nginx' More importantly, here are some variations that we would like to use to show how our resource can be used: ~/chef-repo/cookbooks/ec_nginx/test/fixtures/cookbooks/nginx_test/recipes/default.rb nginx_server 'www.example.com' nginx_server 'notes' do host 'notes.example.com' static_path '/var/www/notes/' landing_page_content '<h1>Read Your Notes</h1>' end nginx_server 'blog' do action :delete end To ensure that chefspec & berkshelf pick up this test cookbook, we're going to add a new group to the Berksfile: ~/chef-repo/cookbooks/ec_nginx/Berksfile # frozen_string_literal: true source 'https://supermarket.chef.io' metadata group :integration do cookbook 'nginx_test', path: 'test/fixtures/cookbooks/nginx_test' end We also need to run berks install: [workstation] ec_nginx $ berks install ... Finally, let's write some tests: ~/chef-repo/cookbooks/ec_nginx/spec/unit/resources/server_spec.rb require 'spec_helper' describe 'nginx_test::default' do context 'on CentOS' do let(:chef_run) do ChefSpec::SoloRunner.new( platform: 'centos', version: '7.4.1708', step_into: ['nginx_server'] ).converge(described_recipe) end it 'converges successfully' do expect { chef_run }.to_not raise_error end it 'installs NGINX' do expect(chef_run).to install_package('nginx') end it 'creates the /var/www/www.example.com/ directory' do expect(chef_run).to create_directory('/var/www/www.example.com') end it 'creates a server file for www.example.com' do expect(chef_run).to render_file('/etc/nginx/conf.d/www.example.com.conf') end end end To run the tests, we'll use chef exec rspec. We'll see this error for each test, but that's a good thing! [workstation] ec_nginx $ chef exec rspec ... NoMethodError: undefined method `nginx_server' for cookbook: nginx_test, recipe: default :Chef::Recipe # /tmp/d20180824-3102-15c0dhg/cookbooks/nginx_test/recipes/default.rb:1:in `from_file' # ./spec/unit/resources/server_spec.rb:10:in `block (3 levels) in <top (required)>' # ./spec/unit/resources/server_spec.rb:26:in `block (3 levels) in <top (required)>' ... Generating the Resource To create our resource we're going to need a /resources directory in our cookbook, but thankfully the Chef CLI has a generator for resources: [workstation] ec_nginx $ chef generate resource server Recipe: code_generator::resource * directory[/home/user/chef-repo/cookbooks/ec_nginx/resources] action create - create new directory /home/user/chef-repo/cookbooks/ec_nginx/resources - restore selinux security context * template[/home/user/chef-repo/cookbooks/ec_nginx/resources/server.rb] action create - create new file /home/user/chef-repo/cookbooks/ec_nginx/resources/server.rb - update content in file /home/user/chef-repo/cookbooks/ec_nginx/resources/server.rb from none to 17d4c4 (diff output suppressed by config) - restore selinux security context This doesn't really provide us a lot out of the gate, just a comment pointing towards the documentation: ~/chef-repo/cookbooks/ec_nginx/resources/server.rb # To learn more about Custom Resources, see https://docs.chef.io/custom_resources.html By default, our custom resource is going to be named as a combination of the cookbook's name (ec_nginx in this case) and the resource's file name. We currently have ec_nginx_server instead of nginx_server. To move our tests forward we're going to need to change that using the resource_name method: ~/chef-repo/cookbooks/ec_nginx/resources/server.rb resource_name :nginx_server Running the tests again, we'll now see different errors: [workstation] ec_nginx $ chef exec rspec ... NoMethodError: undefined method `host' for Custom resource ec_nginx_server from cookbook ec_nginx ... We've moved onto a different error though we don't quite know where host is coming from just yet. Continued in Part 2

Creating Custom Resources - Part 2

00:12:20

Lesson Description:

In this lesson, we're going to continue implementing our own custom resource to allow us to easily specify NGINX configuration to deploy new virtual hosts. Documentation For This Video Chef Custom Resources DocumentationChef Custom Resource DSLDefining custom resource propertiesDefining custom resource actions Adding Parameters and Actions to the Custom Resource When we ended the previous lesson we were left with the following error: [workstation] ec_nginx $ chef exec rspec ... NoMethodError: undefined method `host' for Custom resource ec_nginx_server from cookbook ec_nginx ... This is a bit of an odd error, but it is caused by how we're using the resource in our example cookbook. Specifically, in the properties in the second usage: ~/chef-repo/cookbooks/ec_nginx/test/fixtures/cookbooks/nginx_test/recipes/default.rb nginx_server 'www.example.com' nginx_server 'notes' do host 'notes.example.com' static_path '/var/www/notes/' landing_page_content '<h1>Read Your Notes</h1>' end nginx_server 'blog' do action :delete end The properties that we interact with are Ruby methods, and they are implemented by Chef when we utilize the property method inside of our custom resource. Let's take a small step and only implement the host property: ~/chef-repo/cookbooks/ec_nginx/resources/server.rb resource_name :nginx_server property :host, String We need to specify what the property name is and the type(s) that it can be. In the case of :host, we're expecting it to always be a string, so we'll use the Ruby String class. Rerunning the test we will now see this error: [ workstation] ec_nginx $ chef exec rspec ... NoMethodError: undefined method `static_path' for Custom resource ec_nginx_server from cookbook ec_nginx ... This is another property, so let's implement all of them before running the tests again: ~/chef-repo/cookbooks/ec_nginx/resources/server.rb resource_name :nginx_server property :host, String property :static_path, String property :landing_page_content, String This is the simplest implementation for each property, and we'll modify this as our tests drive us to handle edge cases. Here's what we get when we run the tests again: [workstation] ec_nginx $ chef exec rspec ... Chef::Exceptions::ValidationFailed: Option action must be equal to one of: nothing! You passed :delete. ... Now we're getting somewhere! Every resource shared the action of :nothing, but in our recipe, we're trying to call the action of :delete. We're ready to define our first action. To define an action we'll use the action method. Let's implement the :delete action now: ~/chef-repo/cookbooks/ec_nginx/resources/server.rb resource_name :nginx_server property :host, String property :static_path, String property :landing_page_content, String action :delete do end Not the most interesting implementation, but the tests haven't shown us that we need to implement more. Running the tests again gives us: [workstation] ec_nginx $ chef exec rspec .....FFF Failures: 1) nginx_test::default on CentOS installs NGINX Failure/Error: expect(chef_run).to install_package('nginx') expected "package[nginx]" with action :install to be in Chef run. Other package resources: # ./spec/unit/resources/server_spec.rb:18:in `block (3 levels) in <top (required)>' 2) nginx_test::default on CentOS creates the /var/www/www.example.com/ directory Failure/Error: expect(chef_run).to create_directory('/var/www/www.example.com') expected "directory[/var/www/www.example.com]" with action :create to be in Chef run. Othe$ directory resources: # ./spec/unit/resources/server_spec.rb:22:in `block (3 levels) in <top (required)>' 3) nginx_test::default on CentOS creates a server file for www.example.com Failure/Error: expect(chef_run).to render_file('/etc/nginx/conf.d/www.example.com.conf') expected Chef run to render "/etc/nginx/conf.d/www.example.com.conf" # ./spec/unit/resources/server_spec.rb:26:in `block (3 levels) in <top (required)>' Finished in 1.38 seconds (files took 1.84 seconds to load) 8 examples, 3 failures Now we're seeing some much more specific errors that will lead us to drive the implementation of our default action (that we'll call :create). It does show that we're not actually testing the :delete action at all, but we'll get there. Implementing the :create Action We've set up the boilerplate for our custom resource, based on some testing feedback, and we're ready to get these tests passing. We need to potentially install software and create some directories and files, based on the information that we've been given, so let's do that now: ~/chef-repo/cookbooks/ec_nginx/resources/server.rb resource_name :nginx_server property :host, String, name_property: true property :static_path, String property :landing_page_content, String action :create do package 'nginx' service 'nginx' do action [:start, :enable] end static_path = new_resource.static_path || "/var/www/#{new_resource.host}" directory static_path do recursive true end template "/etc/nginx/conf.d/#{new_resource.host}.conf" do source "nginx-server.conf.erb" variables( host: new_resource.host, static_path: static_path ) notifies :reload, 'service[nginx]' end if new_resource.landing_page_content file "#{static_path}/index.html" do content new_resource.landing_page_content end end end action :delete do end There's quite a bit going on here, so let's break it down: We set the property of :host to match the default name property, unless manually specified using the name_property: true option.We install NGINX and start the service the same way we would in a normal recipe.We create the variable static_path that is the static_path property if specified, or default to a directory based on host within /var/www.The directory to hold static assets is created using the static_path variable.A new NGINX configuration file is added, based on the host value that renders a server configuration template.If the landing_page_content property is set, we make sure that the index.html file for the static_path exists and has the proper content. We're not quite done yet, we'll need to generate the template that we're going to work with: [workstation] ec_nginx $ chef generate template nginx-server.conf Recipe: code_generator::template * directory[/home/user/chef-repo/cookbooks/ec_nginx/templates] action create - create new directory /home/user/chef-repo/cookbooks/ec_nginx/templates - restore selinux security context * template[/home/user/chef-repo/cookbooks/ec_nginx/templates/nginx-server.conf.erb] action create - create new file /home/user/chef-repo/cookbooks/ec_nginx/templates/nginx-server.conf.erb - update content in file /home/user/chef-repo/cookbooks/ec_nginx/templates/nginx-server.conf.erb from none to e3b0c4 (diff output suppressed by config) - restore selinux security context Here's what that template will look like: chef-repo/cookbooks/ec_nginx/templates/nginx-server.conf.erb server { listen 80; listen [::]:80; server_name <%= @host %>; root <%= @static_path %>; location / { } } Finally, let's run our tests: [workstation] ec_nginx $ chef exec rspec ........ Finished in 1.48 seconds (files took 1.83 seconds to load) 8 examples, 0 failures There aren't currently any tests that exercise the :delete action, or the landing_page_content property, so I challenge you to try to implement those yourself before moving on. Spoiler: Example Implementation of :delete Action If you'd like to see an example implementation of the :delete action, here's a simple one with some tests to accompany it: ~/chef-repo/cookbooks/ec_nginx/resources/server.rb # omitted earlier code action :delete do static_path = new_resource.static_path || "/var/www/#{new_resource.host}" directory static_path do recursive true action :delete end file "/etc/nginx/conf.d/#{new_resource.host}.conf" do action :delete end end ~/chef-repo/cookbooks/ec_nginx/spec/unit/resources/server_spec.rb # omitted other tests it 'deletes the host configuration file' do expect(chef_run).to delete_file('/etc/nginx/conf.d/blog.conf') expect(chef_run).to delete_file('/etc/nginx/conf.d/status.example.com.conf') end it 'deletes the static path directory' do expect(chef_run).to delete_directory('/var/www/blog') expect(chef_run).to delete_directory('/var/www/status') end I also modified the nginx_test::default recipe to exercise more of the nginx_server resource's API: ~/chef-repo/cookbooks/ec_nginx/test/fixtures/cookbooks/nginx_test/recipes/default.rb nginx_server 'www.example.com' nginx_server 'notes' do host 'notes.example.com' static_path '/var/www/notes' landing_page_content '<h1>Read your notes!</h1>' end nginx_server 'blog' do action :delete end nginx_server 'status_site' do host 'status.example.com' static_path '/var/www/status' action :delete end

Sharing Custom Resources

00:09:00

Lesson Description:

Now that we have our own custom resource, we're ready to share it so that it can be used by other cookbooks. Thankfully, we already know how to do everything necessary in order to share our custom resource, because we only need to share the cookbook itself. Documentation For This Video Chef Custom Resources DocumentationChef Custom Resource DSL Sharing Our Custom Resource To share the nginx_server resource that we've written, we need to share the entire cookbook that contains it (in this case ec_nginx). Up to this point, we've used cookbooks that are on the public Supermarket, but we can also make other cookbooks that rely on our own cookbooks. To demonstrate this, let's create a cookbook for a specific web application that we'd like to deploy that uses the nginx_server resource. We'll call this cookbook ec_status_site: [workstation] chef-repo $ chef generate cookbook cookbooks/ec_status_site ... [workstation] chef-repo $ cd cookbooks/ec_status_site From here, we can specify that we do have a requirement on the ec_nginx cookbook in our metadata.rb: ~/chef-repo/cookbooks/ec_status_site/metadata.rb name 'ec_status_site' maintainer 'The Authors' maintainer_email 'you@example.com' license 'All Rights Reserved' description 'Installs/Configures ec_status_site' long_description 'Installs/Configures ec_status_site' version '0.1.0' chef_version '>= 12.14' if respond_to?(:chef_version) depends 'ec_nginx' If we run berks install like we have in the past, we'll receive an error: [workstation] ec_status_site $ berks install Resolving cookbook dependencies... Fetching 'ec_status_site' from source at . Fetching cookbook index from https://supermarket.chef.io... Unable to satisfy constraints on package ec_nginx, which does not exist, due to solution constraint (ec_status_site = 0.1.0). Solution constraints that may result in a constraint on ec_nginx: [(ec_status_site = 0.1.0) -> (ec_nginx >= 0.0.0)] Missing artifacts: ec_nginx Demand that cannot be met: (ec_status_site = 0.1.0) Unable to find a solution for demands: ec_status_site (0.1.0) This error is caused by our cookbook not being on the supermarket. The depends line tells Chef that we need the cookbook on the Chef Server, but Berkshelf is the tool that we've been using to get them in the first place. We'll need to be a little more specific to make Berkshelf do the right thing. From here, we can modify our Berksfile so that we use the local cookbook: ~/chef-repo/cookbooks/ec_status_site/Berksfile # frozen_string_literal: true source 'https://supermarket.chef.io' metadata cookbook 'ec_nginx', path: '../ec_nginx' With this change we've fixed the issues when we run berks install: [workstation] ec_status_site $ berks install Resolving cookbook dependencies... Fetching 'ec_nginx' from source at ../ec_nginx Fetching 'ec_status_site' from source at . Using ec_nginx (0.1.0) from source at ../ec_nginx Using ohai (5.2.3) Using ec_status_site (0.1.0) from source at . Our new ec_status_site cookbook allows us to specify everything that we need to run a status site on a server. Let's define the default recipe: ~/chef-repo/cookbooks/ec_status_site/recipes/default.rb nginx_server 'status.example.com' do landing_page_content '<h1>Everything is Great!</h1>' end Finally, let's upload this new cookbook: [workstation] ec_status_site $ berks upload Skipping ec_nginx (0.1.0) (frozen) /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/ridley-5.1.1/lib/ridley/client.rb:271:in `server_url': Ridley::Client#url_prefix at /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/ridley-5.1.1/lib/ridley/client.rb:79 forwarding to private method Celluloid::PoolManager#url_prefix Uploaded ec_status_site (0.1.0) to: 'https://keiththomps1.mylabserver.com:443/organizations/linuxacademy' Skipping ohai (5.2.3) (frozen) Modifying Our Run-list Since our ec_nginx cookbook now provides a resource instead of just recipes for installing NGINX, we likely won’t use it directly in a run-list anymore. Let's remove ec_nginx from the web-node1 run-list and instead use ec_status_site: [workstation] ec_status_site $ knife node show web-node1 -a run_list web-node1: run_list: recipe[ec_base], recipe[ec_nginx::default], recipe[ec_nginx::ohai] [workstation] ec_status_site $ knife node run_list set web-node1 'recipe[ec_base],recipe[ec_nginx::ohai],recipe[ec_status_site::default]' web-node1: run_list: recipe[ec_base] recipe[ec_nginx::ohai] recipe[ec_status_site::default] [workstation] ec_status_site $ We still want the Ohai information about NGINX, so we've left the ec_nginx::ohai recipe in the run-list. As we always do, we need to run chef-client now: [workstation] ec_status_site $ knife ssh 'name:web-node1' 'sudo chef-client' ... servername.mylabserver.com Chef::Exceptions::FileNotFound servername.mylabserver.com ------------------------------ servername.mylabserver.com template[/etc/nginx/conf.d/status.example.com.conf] (/var/chef/ca che/cookbooks/ec_nginx/resources/server.rb line 20) had an error: Chef::Exceptions::FileNotFound: Cookbook 'ec_status_site' (0.1.0) does not contain a file at any of these locations: servername.mylabserver.com templates/centos-7.5.1804/nginx-server.conf.erb servername.mylabserver.com templates/centos/nginx-server.conf.erb servername.mylabserver.com templates/default/nginx-server.conf.erb servername.mylabserver.com templates/nginx-server.conf.erb servername.mylabserver.com servername.mylabserver.com Resource Declaration: servername.mylabserver.com --------------------- servername.mylabserver.com # In /var/chef/cache/cookbooks/ec_status_site/recipes/default.rb servername.mylabserver.com servername.mylabserver.com 1: nginx_server 'status.example.com' do servername.mylabserver.com 2: landing_page_content '<h1>Everything is Great!</h1>' servername.mylabserver.com 3: end servername.mylabserver.com servername.mylabserver.com Compiled Resource: servername.mylabserver.com ------------------ servername.mylabserver.com # Declared in /var/chef/cache/cookbooks/ec_status_site/recipes/de fault.rb:1:in `from_file' servername.mylabserver.com servername.mylabserver.com nginx_server("status.example.com") do servername.mylabserver.com action [:create] servername.mylabserver.com default_guard_interpreter :default servername.mylabserver.com declared_type :nginx_server servername.mylabserver.com cookbook_name "ec_status_site" servername.mylabserver.com recipe_name "default" servername.mylabserver.com landing_page_content "<h1>Everything is Great!</h1>" servername.mylabserver.com host "status.example.com" servername.mylabserver.com end This is unfortunate, but it does show us something that's important to know. Resources like cookbook_file and template assume that they should be looking for the file within the cookbook containing the resource, not the custom resource definition. To fix this, we'll need to make our custom resource definition accept a cookbook name that defaults to itself. Doing this will help make our resource more flexible, allowing the consumer to override the template by creating a nginx-server.conf.erb within the cookbook using the nginx_server resource. Let's modify our resource now: ~/chef-repo/cookbooks/ec_nginx/resources/server.rb resource_name :nginx_server property :host, String, name_property: true property :static_path, String property :landing_page_content, String property :cookbook, String, default: 'ec_nginx' action :create do package 'nginx' service 'nginx' do action [:start, :enable] end static_path = new_resource.static_path || "/var/www/#{new_resource.host}" directory static_path do recursive true end template "/etc/nginx/conf.d/#{new_resource.host}.conf" do source "nginx-server.conf.erb" cookbook new_resource.cookbook variables( host: new_resource.host, static_path: static_path ) notifies :reload, 'service[nginx]' end if new_resource.landing_page_content file "#{static_path}/index.html" do content new_resource.landing_page_content end end end action :delete do end We will need to update the cookbook version with this change: ~/chef-repo/cookbooks/ec_nginx/metadata.rb name 'ec_nginx' maintainer 'John Doe' maintainer_email 'johndoe@example.com' license 'All Rights Reserved' description 'Installs/Configures NGINX' long_description 'Installs/Configures NGINX' version '0.1.1' chef_version '>= 12.14' if respond_to?(:chef_version) depends 'ohai', '~> 5.2.3' Finally, let's upload the updated cookbook and run chef-client again: [workstation] ec_nginx $ berks upload ... [workstation] ec_nginx $ knife ssh 'name:web-node1' 'sudo chef-client' ... To test this, we can curl the server from our workstation with the proper header set: [workstation] ec_nginx $ curl servername.mylabserver.com --header "Host: status.example.com" <h1>Everything is Great!</h1>[workstation] ec_nginx $

Hands-on Labs are real live environments that put you in a real scenario to practice what you have learned without any other extra charge or account to manage.

01:00:00

Custom Resource Concepts

Default Actions and Property Validators

00:12:58

Lesson Description:

We have a basic idea of how to create and use custom resources, but there are some more advanced features that we should be aware of. In this lesson, we'll cover parameter validation and the default action. Documentation For This Video Chef Custom Resources DocumentationChef Custom Resource DSL Explicit default_action In our previous example we've allowed the default action to be implicitly set, but now we need to know how the default action is determined and how to manually set it. To dig into these features we're going to create a cookbook and resource to experiment with: [workstation] chef-repo $ chef generate cookbook cookbooks/exploring_resources ... [workstation] chef-repo $ cd cookbooks/exploring_resources [workstation] exploring_resources $ chef generate resource json_config The current name of exploring_resources_json_config is a little long, so we'll rename this to simple json_config, but the purpose of this resource is to take content a Ruby Hash or String and a path and write out the Hash as a JSON file. Let's set the name, properties, and add actions for :delete, and :create: ~/chef-repo/cookbooks/exploring_resources/resources/json_config.rb resource_name :json_config property :path, String, name_property: true property :content, [Hash, String] action :delete do puts "nnnWe're DELETING the confignnn" end action :create do puts "nnnWe're CREATING the confignnn" end Now we're going to adjust our default recipe to use this resource: ~/chef-repo/cookbooks/exploring_resources/recipes/default.rb json_config "/home/user/chef-repo/cookbooks/exploring_resources/example.json" do content { testing: "this is a test", others: [1, 2, 3, 4] } end Let's see what the default action is currently by running this with chef in local-mode: [workstation] exploring_resources $ chef-client --local-mode -r exploring_resources Starting Chef Client, version 13.10.4 resolving cookbooks for run list: ["exploring_resources"] Synchronizing Cookbooks: - exploring_resources (0.1.0) Installing Cookbook Gems: Compiling Cookbooks... Converging 1 resources Recipe: exploring_resources::default * json_config[/home/user/chef-repo/cookbooks/exploring_resources/example.json] action delete We're DELETING the config (up to date) Running handlers: Running handlers complete Chef Client finished, 0/1 resources updated in 02 seconds We probably don't want :delete to be the default action, but it is because we've defined it first. If we still prefer this ordering for the actions in our resource file then we can explicitly set the default action using the default_action method. Let's set the default action to be :create: ~/chef-repo/cookbooks/exploring_resources/resources/json_config.rb resource_name :json_config property :path, String, name_property: true property :content, [Hash, String] default_action :create action :delete do puts "nnnWe're DELETING the confignnn" end action :create do puts "nnnWe're CREATING the config with #{new_resource.content}nnn" end Running chef-client again we can see that without an explicit action we're now running :create: [workstation] exploring_resources $ chef-client --local-mode -r exploring_resources Starting Chef Client, version 13.10.4 resolving cookbooks for run list: ["exploring_resources"] Synchronizing Cookbooks: - exploring_resources (0.1.0) Installing Cookbook Gems: Compiling Cookbooks... Converging 1 resources Recipe: exploring_resources::default * json_config[/home/user/chef-repo/cookbooks/exploring_resources/example.json] action create We're CREATING the config with {:testing=>"this is a test", :others=>[1, 2, 3, 4]} (up to date) Running handlers: Running handlers complete Chef Client finished, 0/1 resources updated in 02 seconds Validating Parameters The next thing that we'd like to with our new resource is to ensure that the parameter values are valid. There are a few different ways that we can do this as can be found in the documentation. Since our :content parameter can be either a String or a Hash we need to validate that it is valid JSON when it is a string. To do this, we're going to using the callbacks validator and the required validation since we must always have content. Before we implement any of the validations though, let's add another resource to the recipe that doesn't have content at all: ~/chef-repo/cookbooks/exploring_resources/recipes/default.rb json_config "/home/user/chef-repo/cookbooks/exploring_resources/example.json" do content({ testing: "this is a test", others: [1, 2, 3, 4] }) end json_config "/home/user/chef-repo/cookbooks/exploring_resources/empty.json" Running the chef-client again, we will see that it does run currently although we didn't specify content. Let's add the required validation to the parameter: ~/chef-repo/cookbooks/exploring_resources/resources/json_config.rb resource_name :json_config property :path, String, name_property: true property :content, [Hash, String], required: true default_action :create action :delete do puts "nnnWe're DELETING the confignnn" end action :create do puts "nnnWe're CREATING the config with #{new_resource.content}nnn" end Now if we run this again, we will get an error stating that the field is required. [workstation] exploring_resources $ chef-client --local-mode -r exploring_resources Starting Chef Client, version 13.10.4 resolving cookbooks for run list: ["exploring_resources"] Synchronizing Cookbooks: - exploring_resources (0.1.0) Installing Cookbook Gems: Compiling Cookbooks... Converging 2 resources Recipe: exploring_resources::default * json_config[/home/user/chef-repo/cookbooks/exploring_resources/example.json] action create We're CREATING the config with {:testing=>"this is a test", :others=>[1, 2, 3, 4]} (up to date) * json_config[/home/user/chef-repo/cookbooks/exploring_resources/empty.json] action create ================================================================================ Error executing action `create` on resource 'json_config[/home/user/chef-repo/cookbooks/explor ing_resources/empty.json]' ================================================================================ Chef::Exceptions::ValidationFailed ---------------------------------- content is a required property Cookbook Trace: --------------- /home/user/.chef/local-mode-cache/cache/cookbooks/exploring_resources/resources/json_config.rb :13:in `block in class_from_file' Resource Declaration: --------------------- # In /home/user/.chef/local-mode-cache/cache/cookbooks/exploring_resources/recipes/default.rb 5: json_config "/home/user/chef-repo/cookbooks/exploring_resources/empty.json" ... One thing to note is that this error happens during the run phase and not the compilation phase of the chef-client run. At the very least, this provides a better experience for users of this resource so that they can see what was done wrong. The other validation that we wanted to set was that if the value is a String, then it needs to be valid JSON. We need to use our resource a few different ways to demonstrate this: ~/chef-repo/cookbooks/exploring_resources/recipes/default.rb json_config "/home/user/chef-repo/cookbooks/exploring_resources/example.json" do content({ testing: "this is a test", others: [1, 2, 3, 4] }) end json_config "/home/user/chef-repo/cookbooks/exploring_resources/deleted.json" do action :delete end json_config "/home/user/chef-repo/cookbooks/exploring_resources/valid.json" do content '{"foo": "bar", "baz": 1}' end json_config "/home/user/chef-repo/cookbooks/exploring_resources/invalid.json" do content "NOT VALID JSON" end If we run chef-client again we won't see any errors, but now we want to add our new validation: ~/chef-repo/cookbooks/exploring_resources/resources/json_config.rb resource_name :json_config property :path, String, name_property: true property :content, [Hash, String], required: true, callbacks: { 'should be valid JSON' => lambda { |content| return true if content.is_a?(Hash) begin JSON.parse(content) rescue false end } } default_action :create action :delete do puts "nnnWe're DELETING the confignnn" end action :create do puts "nnnWe're CREATING the config with #{new_resource.content}nnn" end The callbacks validator takes a Hash where each key is the validation message and the value is a lambda that returns a value, if the value is "falsy" then the validation fails. We don't need to parse as JSON if we know that the content is a Hash, but if it is a String, then we want to use JSON.parse. This method will raise an error if the String is not valid JSON, so we rescue and explicitly return false. Once again, let's run our recipe again: [workstation] exploring_resources $ chef-client --local-mode -r exploring_resources ... Chef::Exceptions::ValidationFailed ---------------------------------- Option content's value NOT VALID JSON should be valid JSON! Cookbook Trace: --------------- /home/user/.chef/local-mode-cache/cache/cookbooks/exploring_resources/recipes/default.rb:14:in $ block in from_file' /home/user/.chef/local-mode-cache/cache/cookbooks/exploring_resources/recipes/default.rb:13:in $ from_file' Relevant File Content: ---------------------- /home/user/.chef/local-mode-cache/cache/cookbooks/exploring_resources/recipes/default.rb: 7: end 8: 9: json_config "/home/user/chef-repo/cookbooks/exploring_resources/valid.json" do 10: content '{"foo": "bar", "baz": 1}' 11: end 12: 13: json_config "/home/user/chef-repo/cookbooks/exploring_resources/invalid.json" do 14>> content "NOT VALID JSON" 15: end 16: ... This time around, the error happens in the compilation phase instead of the run phase. With this callback we've now added enough validations to mostly ensure that the content that we're going to write is actually JSON compatible (we can easily turn a Hash into JSON).

Idempotency

00:16:07

Lesson Description:

Continuing with more in-depth custom resource concepts, we'll cover how to implement idempotency for entirely custom resources. Documentation For This Video Chef Custom Resources DocumentationChef Custom Resource DSL Idempotency for Custom Resources Our current custom nginx_server resource works by wrapping existing resources that are already idempotent so we don't need to manually do anything, but occasionally, we'll want to create a much more custom resource that is built out of pure Ruby. In that case, we'll need to add some logic around whether or not the resource should run so that it works like all of the built-in resources and we won't waste the effort if nothing needs to change. To accomplish this, the custom resource DSL provides us with two important methods: load_current_value - We explicitly populate a version of the resource with the information we can access on the node.converge_if_changed - The attributes set in load_current_value will be compared with the values in the new_resource that we're processing. If there is a difference then the code within the block will execute. We're going to continue working on the json_config resource and implement it without using any of the built-in Chef resources (although we could definitely accomplish this using built-in resources). Most of what we're going to be doing is going to require us to read/write/delete files, create directories, and parse/load JSON. Writing Actual JSON Before we can make our resource idempotent we should probably make it actually write the file to disk. Here's a pretty straight-forward implementation: ~/chef-repo/cookbooks/exploring_resources/resources/json_config.rb resource_name :json_config property :path, String, name_property: true property :content, [Hash, String], required: true, callbacks: { 'should be valid JSON' => lambda { |content| return true if content.is_a?(Hash) begin JSON.parse(content) rescue false end } } default_action :create action :delete do if ::File.exists?(new_resource.path) ::FileUtils.rm(new_resource.path) end end action :create do file = ::File.open(new_resource.path, 'w') if new_resource.content.is_a?(String) file.write(new_resource.content) else file.write(new_resource.content.to_json) end end Now let's see what happens when we run this: [workstation] exploring_resources $ chef-client --local-mode -r exploring_resources Starting Chef Client, version 13.10.4 resolving cookbooks for run list: ["exploring_resources"] Synchronizing Cookbooks: - exploring_resources (0.1.0) Installing Cookbook Gems: Compiling Cookbooks... Converging 3 resources Recipe: exploring_resources::default * json_config[/home/user/chef-repo/cookbooks/exploring_resources/example.json] action create (up to date) * json_config[/home/user/chef-repo/cookbooks/exploring_resources/empty.json] action delete (up to date) * json_config[/home/user/chef-repo/cookbooks/exploring_resources/valid.json] action create (up to date) Running handlers: Running handlers complete Chef Client finished, 0/3 resources updated in 02 seconds Notice that it says that none of the resources were run, but if we look in our current directory (using ls) we will see that the JSON files were created: [workstation] exploring_resources $ ls Berksfile example.json metadata.rb recipes spec valid.json chefignore LICENSE README.md resources test This is another benefit of adding idempotency to our resources, they will properly report information to the Chef Client. Populating Current Value The first step in making a resource idempotent is to check the current state of the resource on the system using load_current_value. In our case, we want to read in the file's content on disk (if it exists). ~/chef-repo/cookbooks/exploring_resources/resources/json_config.rb resource_name :json_config property :path, String, name_property: true property :content, [Hash, String], required: true, callbacks: { 'should be valid JSON' => lambda { |content| return true if content.is_a?(Hash) begin JSON.parse(content) rescue false end } } default_action :create load_current_value do if ::File.exists?(path) json_file = ::File.open(path, 'r') content JSON.parse(json_file.read) end end action :delete do if ::File.exists?(new_resource.path) converge_by "Deleting file at #{new_resource.path}" do ::FileUtils.rm(new_resource.path) end end end action :create do converge_if_changed do file = ::File.open(new_resource.path, 'w') if new_resource.content.is_a?(String) file.write(new_resource.content) else file.write(new_resource.content.to_json) end end end Now let's create the empty.json file and re-run chef-client: [workstation] exploring_resources $ echo "{}" > empty.json [workstation] exploring_resources $ chef-client --local-mode -r exploring_resources Starting Chef Client, version 13.10.4 resolving cookbooks for run list: ["exploring_resources"] Synchronizing Cookbooks: - exploring_resources (0.1.0) Installing Cookbook Gems: Compiling Cookbooks... Converging 3 resources Recipe: exploring_resources::default * json_config[/home/user/chef-repo/cookbooks/exploring_resources/example.json] action create - update /home/user/chef-repo/cookbooks/exploring_resources/example.json - set content to {:testing=>"this is a test", :others=>[1, 2, 3, 4]} (was {"testing"=>"this is a test", "others"=>[1, 2, 3, 4]}) * json_config[/home/user/chef-repo/cookbooks/exploring_resources/empty.json] action delete - delete /home/user/chef-repo/cookbooks/exploring_resources/empty.json * json_config[/home/user/chef-repo/cookbooks/exploring_resources/valid.json] action create - update /home/user/chef-repo/cookbooks/exploring_resources/valid.json - set content to "{"foo": "bar", "baz": 1}" (was {"foo"=>"bar", "baz"=>1}) Running handlers: Running handlers complete Chef Client finished, 3/3 resources updated in 02 seconds Now we see the familiar output. We need to deal with the :create action having comparison issues, but let's make sure that the :delete action works properly by running this again. [workstation] exploring_resources $ chef-client --local-mode -r exploring_resources Starting Chef Client, version 13.10.4 resolving cookbooks for run list: ["exploring_resources"] Synchronizing Cookbooks: - exploring_resources (0.1.0) Installing Cookbook Gems: Compiling Cookbooks... Converging 3 resources Recipe: exploring_resources::default * json_config[/home/user/chef-repo/cookbooks/exploring_resources/example.json] action create - update /home/user/chef-repo/cookbooks/exploring_resources/example.json - set content to {:testing=>"this is a test", :others=>[1, 2, 3, 4]} (was {"testing"=>"this is a test", "others"=>[1, 2, 3, 4]}) * json_config[/home/user/chef-repo/cookbooks/exploring_resources/empty.json] action delete (up to date) * json_config[/home/user/chef-repo/cookbooks/exploring_resources/valid.json] action create - update /home/user/chef-repo/cookbooks/exploring_resources/valid.json - set content to "{"foo": "bar", "baz": 1}" (was {"foo"=>"bar", "baz"=>1}) Running handlers: Running handlers complete Chef Client finished, 2/3 resources updated in 02 seconds Since the empty.json file was already removed there was nothing for the :delete action to do. The converge_by method allowed us to report the information about the converge to the Chef client when what we were doing wasn't necessarily based on the values set by load_current_value. To fix the :create action, we're going to make sure that whether we're reading in from a file or taking in the new_resource properties that we always have something that will compare properly. We'll use a Mash in this case. ~/chef-repo/cookbooks/exploring_resources/resources/json_config.rb resource_name :json_config property :path, String, name_property: true property :content, [Hash, String], callbacks: { 'should be valid JSON' => lambda { |content| puts "We're running callbacks" return true if content.is_a?(Hash) begin JSON.parse(content) rescue false end } }, coerce: lambda { |content| puts "We're running coerce" return Mash.new(content) if content.is_a?(Hash) begin Mash.new(JSON.parse(content)) rescue content end } # Omitted unchanged implementation ... We've done a few things here. First, we've removed the required: true from the :content property because it will actually cause issues in our :create action where the file doesn't exist yet at all. Second, we've added the coerce option. The coerce option allows us to specify a way to normalize the data passed into our resource and is extremely useful when multiple data types are accepted by a property. We've added some puts statements in here temporarily so that we can the order that the callbacks and coerce are run in. Let's run chef-client again: [workstation] exploring_resources $ chef-client --local-mode -r exploring_resources Starting Chef Client, version 13.10.4 resolving cookbooks for run list: ["exploring_resources"] Synchronizing Cookbooks: - exploring_resources (0.1.0) Installing Cookbook Gems: Compiling Cookbooks... We're running coerce We're running callbacks We're running coerce We're running callbacks Converging 3 resources Recipe: exploring_resources::default * json_config[/home/user/chef-repo/cookbooks/exploring_resources/example.json] action createWe're running coerce We're running callbacks (up to date) * json_config[/home/user/chef-repo/cookbooks/exploring_resources/empty.json] action delete (up to date) * json_config[/home/user/chef-repo/cookbooks/exploring_resources/valid.json] action createWe're running coerce We're running callbacks (up to date) Running handlers: Running handlers complete Chef Client finished, 0/3 resources updated in 02 seconds It looks like coerce is always run first and we'll need to take this into consideration when we're writing custom properties like content. With this information in mind, we can actually simplify our callbacks and load_current_value so that we're only parsing JSON in one spot: ~/chef-repo/cookbooks/exploring_resources/resources/json_config.rb resource_name :json_config property :path, String, name_property: true property :content, [Hash, String], callbacks: { 'should be valid JSON' => lambda { |content| content.is_a?(Mash) } }, coerce: lambda { |content| return Mash.new(content) if content.is_a?(Hash) begin Mash.new(JSON.parse(content)) rescue content end } default_action :create load_current_value do if ::File.exists?(path) json_file = ::File.open(path, 'r') content json_file.read end end # Omitted unchanged implementation ... Let's remove the puts statements and manually change the data in example.json and empty.json to make sure that it does still run if the content is different. [workstation] exploring_resources $ echo "{}" > example.json [workstation] exploring_resources $ echo "{}" > empty.json [workstation] exploring_resources $ chef-client --local-mode -r exploring_resources Starting Chef Client, version 13.10.4 resolving cookbooks for run list: ["exploring_resources"] Synchronizing Cookbooks: - exploring_resources (0.1.0) Installing Cookbook Gems: Compiling Cookbooks... Converging 3 resources Recipe: exploring_resources::default * json_config[/home/user/chef-repo/cookbooks/exploring_resources/example.json] action create - update /home/user/chef-repo/cookbooks/exploring_resources/example.json - set content to {"testing"=>"this is a test", "others"=>[1, 2, 3, 4]} (was {}) * json_config[/home/user/chef-repo/cookbooks/exploring_resources/empty.json] action delete - delete file at /home/user/chef-repo/cookbooks/exploring_resources/empty.json * json_config[/home/user/chef-repo/cookbooks/exploring_resources/valid.json] action create (up to date) Running handlers: Running handlers complete Chef Client finished, 2/3 resources updated in 02 seconds We've successfully created a pure Ruby resource that has idempotency like the built-in resources.

Nested Custom Resources and Resource Collections

Nested Resources

00:08:23

Lesson Description:

Before we wrap up our discussion on resources and custom resources it's worth looking at how resources are grouped and run. In this lesson, we'll take a look at the nesting custom resources. Documentation For This Video Extending Chef Badge Scope What Are Nested Resource Collections? Looking through the scope document for the Extending Chef Exam we'll notice that there is a section called "Nested Resource Collections". Unfortunately, there is no reference to "nested resource collection" or "resource collection" in the Chef documentation. To figure out what is expected of us we need to go digging around a little and try to define the overall subject by laying out the subtopics in the section of the scope doc. Cookbook Setup Up to now, we've created a few different custom resources that either use raw Ruby or Chef's built-in resources, but what happens when we try to use custom resources within a custom resource? To test this out, we're going to create a new cookbook that uses our existing resources and dig into what happens when they are nested. [workstation] chef-repo $ chef generate cookbook cookbooks/dev_setup ... [workstation] chef-repo $ cd cookbooks/dev_setup ... We'll quickly set up our dependencies and our .kitchen.yml file so that we're ready to run tests and explore the interactions of nested resources. ~/chef-repo/cookbooks/dev_setup/metadata.rb name 'dev_setup' maintainer 'The Authors' maintainer_email 'you@example.com' license 'All Rights Reserved' description 'Installs/Configures dev_setup' long_description 'Installs/Configures dev_setup' version '0.1.0' chef_version '>= 12.14' if respond_to?(:chef_version) depends 'exploring_resources' depends 'ec_nginx' ~/chef-repo/cookbooks/dev_setup/Berksfile # frozen_string_literal: true source 'https://supermarket.chef.io' metadata cookbook 'exploring_resources', path: '../exploring_resources' cookbook 'ec_nginx', path: '../ec_nginx' ~/chef-repo/cookbooks/dev_setup/.kitchen.yml --- driver: name: docker use_sudo: false provisioner: name: chef_zero # You may wish to disable always updating cookbooks in CI or other testing environments. # For example: # always_update_cookbooks: <%= !ENV['CI'] %> always_update_cookbooks: true product_name: chef product_version: 13.10.4 verifier: name: inspec platforms: - name: centos-7 driver_config: run_command: /usr/sbin/init privileged: true provision_command: - systemctl enable sshd.service suites: - name: default run_list: - recipe[dev_setup::default] verifier: inspec_tests: - test/integration/default attributes: With those files in place, we're ready to "install" our dependencies and ensure that kitchen is working properly. [workstation] dev_setup $ berks install ... [workstation] dev_setup $ kitchen test Nesting Custom Resources Now that we have a cookbook, we're going to create a new custom resource that leverages both nginx_server and json_config. This resource will be dev_setup_init. Let's generate this resource now: [workstation] dev_setup $ chef generate resource init Recipe: code_generator::resource * directory[/home/user/chef-repo/cookbooks/dev_setup/resources] action create - create new directory /home/user/chef-repo/cookbooks/dev_setup/resources - restore selinux security context * template[/home/user/chef-repo/cookbooks/dev_setup/resources/init.rb] action create - create new file /home/user/chef-repo/cookbooks/dev_setup/resources/init.rb - update content in file /home/user/chef-repo/cookbooks/dev_setup/resources/init.rb from none to 17d4c4 (diff output suppressed by config) - restore selinux security context In this resource, we're going to take in arguments that map directly to the nginx_server and then we'll generate a JSON file that includes some of this configuration information. Here's the implementation for our dev_setup_init resource: ~/chef-repo/cookbooks/dev_setup/resources/init.rb property :host, String, name_property: true property :content, String action :create do nginx_server new_resource.host do if new_resource.content landing_page_content new_resource.content end end directory "/etc/chef/ohai/hints" do recursive true end json_config "/etc/chef/ohai/hints/#{new_resource.host}.json" do content({ host: new_resource.host }) end end We're not really creating a hint with this JSON configuration we just needed somewhere to put the file. Let's change the default recipe and then run kitchen converge to see the Chef output to examine what happens. ~/chef-repo/cookbooks/dev_setup/recipes/default.rb package 'epel-release' dev_setup_init 'www.example.com' Finally, let's run kitchen converge: [workstation] dev_setup $ kitchen converge ... Synchronizing Cookbooks: - exploring_resources (0.1.0) - ohai (5.2.5) - dev_setup (0.1.0) - ec_nginx (0.2.1) Installing Cookbook Gems: Compiling Cookbooks... Converging 2 resources Recipe: dev_setup::default * yum_package[epel-release] action install - install version 7-11 of package epel-release * dev_setup_init[www.example.com] action create * nginx_server[www.example.com] action create * yum_package[nginx] action install - install version 1.12.2-2.el7 of package nginx * service[nginx] action start - start service service[nginx] * service[nginx] action enable - enable service service[nginx] * directory[/var/www/www.example.com] action create - create new directory /var/www/www.example.com * template[/etc/nginx/conf.d/www.example.com.conf] action create - create new file /etc/nginx/conf.d/www.example.com.conf - update content in file /etc/nginx/conf.d/www.example.com.conf from none to e31b09 --- /etc/nginx/conf.d/www.example.com.conf 2018-09-05 14:57:46.939038691 +0000 +++ /etc/nginx/conf.d/.chef-www20180905-243-1evyei2.example.com.conf 2018-09-05 14:57:46.938038698 +0000 @@ -1 +1,11 @@ +server { + listen 80; + listen [::]:80; + server_name www.example.com; + + root /var/www/www.example.com; + + location / { + } +} * service[nginx] action reload - reload service service[nginx] * directory[/etc/chef/ohai/hints] action create - create new directory /etc/chef/ohai/hints * json_config[/etc/chef/ohai/hints/www.example.com.json] action create - update /etc/chef/ohai/hints/www.example.com.json - set content to {"host"=>"www.example.com"} (was nil) Running handlers: Running handlers complete Chef Client finished, 11/11 resources updated in 42 seconds Downloading files from <default-centos-7> Finished converging <default-centos-7> (1m1.69s). ... Chef is showing us the name of each of the resources that we're calling in our recipe, but also the resources that are used by those resources by using some whitespace to nest the calls. This demonstration leads us directly to discussing resource collections.

Resource Collections

00:10:25

Lesson Description:

From what we've seen nesting custom resources works the way we'd probably expect and custom resources gradually expand to call the resources that they rely on, but does it matter at what level a resource is called? In this lesson, we'll look at resource collections to see how resources are grouped and what type of communication we can have between them. Documentation For This Video Extending Chef Badge Scope What Are Resource Collection? Resource collections are groups of resources within a given run_context. For the most part, we're not going to directly manipulate these, but we will want to know where the lines of a collection are defined. To test this out, we're going to add some resources that echo messages at various spots within our recipe and custom resources and attempt to send notifications from various locations. Let's start off by adding an execute resources to the default recipe of our dev_setup cookbook: ~/chef-repo/cookbooks/dev_setup/recipes/default.rb package 'epel-release' dev_setup_init 'www.example.com' execute 'echo-in-parent-recipe' do command 'echo "TRIGGERED IN THE RECIPE"' action :nothing end We'll also change our custom resource to have some notifications within itself and also to the parent recipe: ~/chef-repo/cookbooks/dev_setup/resources/init.rb property :host, String, name_property: true property :content, String action :create do execute 'echo-in-same-resource' do command 'echo "FROM SAME dev_setup_init"' action :nothing end execute 'echo-in-custom-resource' do command 'echo "FROM CHILD dev_setup_init"' action :nothing end nginx_server new_resource.host do if new_resource.content landing_page_content new_resource.content end end directory "/etc/chef/ohai/hints" do recursive true notifies :run, 'execute[echo-in-same-resource]', :immediately end json_config "/etc/chef/ohai/hints/#{new_resource.host}.json" do content({ host: new_resource.host }) notifies :run, 'execute[echo-in-parent-recipe]', :immediately end end Now we'll run kitchen test so that it creates a new container and all of these notifications will be triggered: [workstation] dev_setup $ kitchen test ... resolving cookbooks for run list: ["dev_setup::default"] Synchronizing Cookbooks: - exploring_resources (0.1.0) - ec_nginx (0.2.1) - ohai (5.2.5) - dev_setup (0.1.0) Installing Cookbook Gems: Compiling Cookbooks... Converging 3 resources Recipe: dev_setup::default * yum_package[epel-release] action install - install version 7-11 of package epel-release * dev_setup_init[www.example.com] action create * execute[echo-in-same-resource] action nothing (skipped due to action :nothing) * execute[echo-in-custom-resource] action nothing (skipped due to action :nothing) * nginx_server[www.example.com] action create * yum_package[nginx] action install - install version 1.12.2-2.el7 of package nginx * service[nginx] action start - start service service[nginx] * service[nginx] action enable - enable service service[nginx] * directory[/var/www/www.example.com] action create - create new directory /var/www/www.example.com * template[/etc/nginx/conf.d/www.example.com.conf] action create - create new file /etc/nginx/conf.d/www.example.com.conf - update content in file /etc/nginx/conf.d/www.example.com.conf from none to e31b09 --- /etc/nginx/conf.d/www.example.com.conf 2018-09-05 16:06:20.416324619 +0000 +++ /etc/nginx/conf.d/.chef-www20180905-243-1q0egwr.example.com.conf 2018-09-05 16:06:20.416324619 +0000 @@ -1 +1,11 @@ +server { + listen 80; + listen [::]:80; + server_name www.example.com; + + root /var/www/www.example.com; + + location / { + } +} * service[nginx] action reload - reload service service[nginx] * directory[/etc/chef/ohai/hints] action create - create new directory /etc/chef/ohai/hints * execute[echo-in-same-resource] action run - execute echo "FROM SAME dev_setup_init" * json_config[/etc/chef/ohai/hints/www.example.com.json] action create - update /etc/chef/ohai/hints/www.example.com.json - set content to {"host"=>"www.example.com"} (was nil) * execute[echo-in-parent-recipe] action run - execute echo "TRIGGERED IN THE RECIPE" * execute[echo-in-parent-recipe] action nothing (skipped due to action :nothing) Running handlers: Running handlers complete Chef Client finished, 13/16 resources updated in 45 seconds ... This is showing us that from within a custom resource we can trigger notifications for resources: Defined in the parent recipeDefined within the same resource action What happens when we try to trigger a resource that is in a custom resource from within the parent recipe? Let's change our recipe to attempt this: ~/chef-repo/cookbooks/dev_setup/recipes/default.rb package 'epel-release' dev_setup_init 'www.example.com' execute 'echo-in-parent-recipe' do command 'echo "TRIGGERED IN THE RECIPE"' action :nothing end execute 'run child resource' do command 'echo "Attempt to run child resource"' notifies :run, 'execute[echo-in-custom-resource]', :immediately end Now when we run kitchen test again to see what happens: [workstation] dev_setup $ kitchen test ... [2018-09-05T16:11:13+00:00] ERROR: resource execute[run child resource] is configured to notify resource execute[echo-in-custom-resource] with action run, but execute[echo-in-custom-resource] cannot be found in the resource collection. execute[run child resource] is defined in /tmp/kitchen/cache/cookbooks/dev_setup/recipes/default.rb:10:in `from_file' ... This error shows us that we can't trigger a resource that is defined at a deeper level. This brings us to resource collections. Viewing the Resource Collections Resource collections aren't currently documented very much, but they essentially hold onto the resources that are part of the currently running context (or run_context to be more specific). We can view this anywhere that we can use the recipe DSL using run_context.resource_collection We're going to add a puts statement to the default recipe and within the dev_setup_init action. Additionally, let's see what happens if we try to subscribe to an action within a child resource collection. ~/chef-repo/cookbooks/dev_setup/recipes/default.rb package 'epel-release' dev_setup_init 'www.example.com' execute 'echo-in-parent-recipe' do command 'echo "TRIGGERED IN THE RECIPE"' action :nothing end execute 'echo-in-parent-recipe-subscribe' do command 'echo "TRIGGERED BY SUBSCRIBING TO RESOURCE WITHIN dev_setup_init"' action :nothing subscribes :run, 'directory[/etc/chef/ohai/hints]' end puts "nrecipe resource collection: #{run_context.resource_collection.map { |item| item.name }}n" ~/chef-repo/cookbooks/dev_setup/resources/init.rb property :host, String, name_property: true property :content, String action :create do execute 'echo-in-same-resource' do command 'echo "FROM SAME dev_setup_init"' action :nothing end execute 'echo-in-custom-resource' do command 'echo "FROM CHILD dev_setup_init"' action :nothing end nginx_server new_resource.host do if new_resource.content landing_page_content new_resource.content end end directory "/etc/chef/ohai/hints" do recursive true notifies :run, 'execute[echo-in-same-resource]', :immediately end json_config "/etc/chef/ohai/hints/#{new_resource.host}.json" do content({ host: new_resource.host }) notifies :run, 'execute[echo-in-parent-recipe]', :immediately end puts "ndev_setup_init resource collection: #{run_context.resource_collection.map { |item| item.name }}n" end Let's run kitchen test again to see what is printed out: [workstation] dev_setup $ kitchen test ... Synchronizing Cookbooks: - exploring_resources (0.1.0) - ec_nginx (0.2.1) - dev_setup (0.1.0) - ohai (5.2.5) Installing Cookbook Gems: Compiling Cookbooks... recipe resource collection: ["epel-release", "www.example.com", "echo-in-parent-recipe", "echo-in-parent-recipe-subscribe"] Converging 4 resources Recipe: dev_setup::default * yum_package[epel-release] action install - install version 7-11 of package epel-release * dev_setup_init[www.example.com] action create dev_setup_init resource collection: ["echo-in-same-resource", "echo-in-custom-resource", "www.example.com", "/etc/chef/ohai/hints", "/etc/chef /ohai/hints/www.example.com.json"] * execute[echo-in-same-resource] action nothing (skipped due to action :nothing) * execute[echo-in-custom-resource] action nothing (skipped due to action :nothing) * nginx_server[www.example.com] action create * yum_package[nginx] action install - install version 1.12.2-2.el7 of package nginx * service[nginx] action start - start service service[nginx] * service[nginx] action enable - enable service service[nginx] * directory[/var/www/www.example.com] action create - create new directory /var/www/www.example.com * template[/etc/nginx/conf.d/www.example.com.conf] action create - create new file /etc/nginx/conf.d/www.example.com.conf - update content in file /etc/nginx/conf.d/www.example.com.conf from none to e31b09 --- /etc/nginx/conf.d/www.example.com.conf 2018-09-05 20:36:54.394044815 +0000 +++ /etc/nginx/conf.d/.chef-www20180905-239-1msx6gm.example.com.conf 2018-09-05 20:36:54.394044815 +0000 @@ -1 +1,11 @@ +server { + listen 80; + listen [::]:80; + server_name www.example.com; + + root /var/www/www.example.com; + + location / { + } +} * service[nginx] action reload - reload service service[nginx] * directory[/etc/chef/ohai/hints] action create - create new directory /etc/chef/ohai/hints * execute[echo-in-same-resource] action run - execute echo "FROM SAME dev_setup_init" * json_config[/etc/chef/ohai/hints/www.example.com.json] action create - update /etc/chef/ohai/hints/www.example.com.json - set content to {"host"=>"www.example.com"} (was nil) * execute[echo-in-parent-recipe] action run - execute echo "TRIGGERED IN THE RECIPE" * execute[echo-in-parent-recipe] action nothing (skipped due to action :nothing) * execute[echo-in-parent-recipe-subscribe] action nothing (skipped due to action :nothing) Running handlers: Running handlers complete Chef Client finished, 13/17 resources updated in 39 seconds ... Notice that the parent resources aren't int he resource collection of the dev_setup_init, but because they are within the parent context so we were able to notify them. Our subscription also didn't get triggered although there was a resource named directory[/etc/chef/ohai/hints] that ran because it was not in the same context or a parent context from our recipe.

Chef Handlers

Using Chef Handler DSL

What are Chef Handlers?

00:02:27

Lesson Description:

In addition to customizing how we work with Chef resources, or the information that we store, we occassionally want to customize what happens during a chef-client run. In this section we'll learn how we can utilize Chef Handlers to carry out other actions when specific situations arise during a chef-client run. Documentation For This Video Chef Handlers DocumentationChef Handler DSL DocumentationPDF of Slideshow

Writing Custom Chef Handlers

00:10:09

Lesson Description:

To learn more about Chef handlers, we're going to go ahead and create our own custom handler that will send messages when specific situations arise in our chef-client runs. Documentation For This Video Chef Handlers DocumentationChef Handler DSL DocumentationThe run_status Object Creating a Custom Chef Handler The primary use that we'll find for custom Chef handlers is sending some sort of notification to an outside service. Network communication isn't something that we need to cover in this course, so we're going to instead utilize a simpler example where we can write lines to a custom log file. To get started, let's create a new cookbook to hold onto our custom handler: [workstation] chef-repo $ chef generate cookbook cookbooks/ec_handlers ... [workstation] chef-repo $ cd cookbooks/ec_handlers [workstation] ec_handlers $ Just like we did when we were working with Ohai plugins, we're going to be creating our custom handler as a standard Ruby file in our files directory. This handler will be called json_logger: [workstation] ec_handlers $ chef generate file json_logger.rb ... Our goal here will be to write outlines to the file that are individually parseable JSON objects. To get started we need to create a new subclass of the Chef::Handler class: ~/chef-repo/cookbooks/ec_handlers/files/default/json_logger.rb require 'chef' require 'chef/handler' module EcHandlers class JsonLogger < Chef::Handler def report end end end For the time being, we're going to adjust the report method and utilize the run_status object to determine what file to write to, and what information to write. The run_status object changes as the chef-client run progresses, and the information that is available changes. It doesn't make sense to have the end_time when the run is still in progress. Here's what our implementation is going to look like: ~/chef-repo/cookbooks/ec_handlers/files/default/json_logger.rb require 'chef' require 'chef/handler' module EcHandlers class JsonLogger < Chef::Handler def initialize(log_directory="/var/log") @log_directory = log_directory end def report puts "nRunning JsonLogger with file: #{log_file.path}n" log_file.write({ success: run_status.success?, start_time: run_status.start_time, node_name: run_status.node.name, end_time: run_status.end_time }.to_json + "n") log_file.close() end def log_file file_name = if !run_status.end_time 'chef-start.json' elsif run_status.success? 'chef-report.json' elsif run_status.failed? 'chef-exception.json' end ::File.open("#{@log_directory}/#{file_name}", 'a') end end end Handlers are standard Ruby objects. If we want to have a little more information passed in when we create them, we can do it by defining the initialize method. In this case, we're taking in an optional log_directory and storing it as an instance variable. We've defined a method call log_file that determines what file to write to, then opens that file, and returns it. Finally, in report we're now utilizing the log_file to write some information as JSON to the file. Now that we have a custom handler written, we're ready to learn how we can deploy this to a node and configure it.

Delivering and Configuring Handlers

00:16:02

Lesson Description:

Now that our first custom handler is written, we need to deploy it to a node so we can see it in action. In this lesson, we'll learn how we can deploy and configure custom handlers using the chef_handler cookbook. Documentation For This Video Chef Handlers DocumentationChef Handler DSL DocumentationThe chef_handler Community Cookbook Notes About The chef_handler Cookbook For the purposes of this exam and course, we're still using Chef 13. Because of that, we're encouraged to use the chef_handler cookbook, but you may notice that the page on the Supermarket shows a large deprecation warning. The chef_handler resource that this cookbook provides has been moved into Chef proper, so starting with Chef 14 it is no longer necessary to rely on a cookbook to have access to this resource. Writing Our Recipe We need to download the chef_handler cookbook and utilize it in a recipe so that Chef can easily deploy the handler. Let's add the dependency now: ~/chef-repo/cookbooks/ec_handlers/metadata.rb name 'ec_handlers' maintainer 'The Authors' maintainer_email 'you@example.com' license 'All Rights Reserved' description 'Installs/Configures ec_handlers' long_description 'Installs/Configures ec_handlers' version '0.1.0' chef_version '>= 12.14' if respond_to?(:chef_version) depends 'chef_handler', '~> 3.0.3' Following that with a quick berks install: [workstation] ec_handlers $ berks install ... Now we're ready to add a recipe to configure these. We'll use the default one for now, but as more handlers are written, we'd likely want to split them off into separate recipes. ~/chef-repo/cookbooks/ec_handlers/recipes/default.rb handler_path = "#{Chef::Config[:file_cache_path]}/json_logger.rb" cookbook_file handler_path do source 'json_logger.rb' action :nothing end.run_action(:create) chef_handler 'EcHandlers::JsonLogger' do source handler_path type({ exception: true, start: true, report: true }) action :nothing end.run_action(:enable) Unlike the ohai_plugin resource that we used before, we do need to manually move the json_logger.rb file over, so we put that in the directory Chef::Config[:file_cache_path]. Once we have the file in place, we're able to reference the file using its full path in the chef_handler resource. The type attribute allows us to mention which of the 3 types of handlers we want this to be. In this case, we've set it to be used for all of them. Lastly, we want to make sure that our handler is set up before the chef-client run occurs by using run_action(:create), which will cause these resources to be processed as part of the compile phase. With all of that in place, let's upload our cookbook, add it to the run-list of web-node1, and run chef-client. [workstation] ec_handlers $ berks upload ... [workstation] ec_handlers $ knife node run_list add web-node1 ec_handlers --after 'recipe[ec_base]' web-node1: run_list: recipe[ec_base] recipe[ec_handlers] recipe[ec_nginx::ohai] recipe[ec_status_site::default] [workstation] ec_handlers $ knife ssh 'name:web-node1' 'sudo chef-client' ... servername.mylabserver.com servername.mylabserver.com Running handlers: servername.mylabserver.com servername.mylabserver.com Running JsonLogger with file: /var/log/chef-report.json servername.mylabserver.com - EcHandlers::JsonLogger servername.mylabserver.com Running handlers complete servername.mylabserver.com Chef Client finished, 1/33 resources updated in 07 seconds At the end of the output we ran the handler. Since the run completed successfully, we should see a new file at /var/log/chef-report.json on the node. Here I've ssh'ed to the node directly: [root@servername ~]# cat /var/log/chef-report.json {"success":true,"start_time":"2018-09-12 15:31:08 UTC","node_name":"web-node1","end_time":"2018-09-12 15:31:14 +0000"} It makes sense that we wouldn't have a /var/log/chef-exception.json file because it didn't fail, but why don't we have a /var/log/chef-start.json file? This is because we were setting up the handler during the chef-client run, so we'd need to manually run start handler after we've set it up. If we run chef-client again, we'll still see that the start handler didn't run. Setting Up A Start Handler To set up a start handler, we can do one of two things: Utilize the chef-client::config recipe, setting more attributes to add the start_handlers setup to the client.rb.Manually edit the client.rb file to add a start handler. Utilizing the chef-client cookbook is definitely the best approach for this. It is worth noting that we could avoid using the chef_handler resource entirely by instead using cookbook_file, along with this configuration, in the client.rb for both exception and report type handlers. Let's change our recipe to remove the start type from the chef_handler, since it doesn't work how we would expect. ~/chef-repo/cookbooks/ec_handlers/recipes/default.rb handler_path = "#{Chef::Config[:file_cache_path]}/json_logger.rb" cookbook_file handler_path do source 'json_logger.rb' action :nothing end.run_action(:create) chef_handler 'EcHandlers::JsonLogger' do source handler_path type({ exception: true, report: true }) action :nothing end.run_action(:enable) Next, we'll adjust what we set in the ec_base attributes to add our handler as a start handler. ~/chef-repo/cookbooks/ec_base/attributes/chef-client.rb node.default['ohai']['disabled_plugins'] = [ :Perl, ] node.default['chef_client']['config']['automatic_attribute_blacklist'] = [ 'languages' ] node.default['chef_client']['load_gems']['json_logger'] = { require_name: "#{Chef::Config[:file_cache_path]}/json_logger.rb" } node.default['chef_client']['config']['start_handlers'] = [ { class: 'EcHandlers::JsonLogger', arguments: [] } ] We need to update the version for our cookbooks, and push them up to the server before running chef-client once again. Finally, let's try chef-client again to see if our start handler runs: [workstation] ec_handlers $ knife ssh 'name:web-node1' 'sudo chef-client' ... servername.mylabserver.com * template[/etc/chef/client.rb] action create [29/6404] servername.mylabserver.com - update content in file /etc/chef/client.rb from 34419f to afa5fc servername.mylabserver.com --- /etc/chef/client.rb 2018-09-12 18:20:15.760053479 +0000 servername.mylabserver.com +++ /etc/chef/.chef-client20180912-7012-40op3a.rb 2018-09-12 18:21:01.729868264 +0000 servername.mylabserver.com @@ -1,3 +1,11 @@ servername.mylabserver.com +["/var/chef/cache/json_logger.rb"].each do |lib| servername.mylabserver.com + begin servername.mylabserver.com + require lib servername.mylabserver.com + rescue LoadError servername.mylabserver.com + Chef::Log.warn "Failed to load #{lib}. This should be resolved after a chef run." servername.mylabserver.com + end servername.mylabserver.com +end servername.mylabserver.com + servername.mylabserver.com automatic_attribute_blacklist ["languages"] servername.mylabserver.com chef_server_url "https://keiththomps1.mylabserver.com/organizations/linuxacademy" servername.mylabserver.com client_fork true servername.mylabserver.com @@ -8,4 +16,11 @@ servername.mylabserver.com servername.mylabserver.com servername.mylabserver.com ohai.disabled_plugins = [:Perl] servername.mylabserver.com + servername.mylabserver.com +# Do not crash if a handler is missing / not installed yet servername.mylabserver.com +begin servername.mylabserver.com + start_handlers << EcHandlers::JsonLogger.new() servername.mylabserver.com +rescue NameError => e servername.mylabserver.com + Chef::Log.error e servername.mylabserver.com +end servername.mylabserver.com - restore selinux security context servername.mylabserver.com * ruby_block[reload_client_config] action run servername.mylabserver.com - execute the ruby block reload_client_config ... We can see here that the handler is added in an intelligent way, so that it won't crash if we made a typo. While the start handler still didn't run this time, it will run on subsequent runs.

The Chef Handler DSL

00:08:38

Lesson Description:

Now that we know a few different ways to set up Chef Handlers, we're ready to look into how we can also handle events that don't fall under the start, report, or exception categories by using the Chef Handler DSL. Documentation For This Video Chef Handlers DocumentationChef Handler DSL DocumentationChef Handler Event Types Registering a Handler For an Event The list of events that can be tied into this is pretty long, and we can do it by utilizing the Chef Handler DSL from within a recipe. For our purposes, we're going to add a little more to the default recipe of our ec_handlers cookbook: ~/chef-repo/cookbooks/ec_handlers/recipes/default.rb handler_path = "#{Chef::Config[:file_cache_path]}/json_logger.rb" cookbook_file handler_path do source 'json_logger.rb' action :nothing end.run_action(:create) chef_handler 'EcHandlers::JsonLogger' do source handler_path type({ exception: true, start: true, report: true }) action :nothing end.run_action(:enable) Chef.event_handler do on :resource_skipped do |run_context| puts "Resource Skipped for #{run_context.node.name}" end on :ohai_completed do puts "nOhai completed!!n" end end The event_handler method on the Chef object provides us with a DSL for listening to events using the on method. From there, we register a handler for the :resource_skipped and :ohai_completed events. The contents of these blocks are essentially the same sort of thing that we did in our report function, where we created a handler object. Let's update the version of the cookbook, upload it, and re-run chef-client: [workstation] ec_handlers $ vim metadata.rb # Change version number ... [workstation] ec_handlers $ berks upload ... [workstation] ec_handlers $ knife ssh 'name:web-node1' 'sudo chef-client' ... servername.mylabserver.com Resource Skipped for web-node1 servername.mylabserver.com * cookbook_file[/etc/chef/ohai/plugins/nginx.rb] action create (up to date) ... All throughout the run, we can see the Resource Skipped message anytime the resource was considered "up to date," but we don't see our Ohai completed message anywhere. This is because the :ohai_completed event occurs before the recipe has been processed and the event handler added. Extending /etc/chef/client.rb With Additional Ruby Files To tie into some events we need to make sure that our code is registered very early and since our /etc/chef/client.rb is being managed by the chef-client cookbook we can leverage the fact that it will read in additional files from the /etc/chef/client.d/ directory. Let's move our handlers into a cookbook file called event_handlers.rb and place that file into this directory: ~/chef-repo/cookbooks/ec_handlers/files/default/event_handlers.rb Chef.event_handler do on :resource_skipped do |run_context| puts "Resource Skipped for #{run_context.node.name}" end on :ohai_completed do puts "nOhai completed!!n" end end ~/chef-repo/cookbooks/ec_handlers/recipes/default.rb handler_path = "#{Chef::Config[:file_cache_path]}/json_logger.rb" cookbook_file handler_path do source 'json_logger.rb' action :nothing end.run_action(:create) chef_handler 'EcHandlers::JsonLogger' do source handler_path type({ exception: true, start: true, report: true }) action :nothing end.run_action(:enable) cookbook_file '/etc/chef/client.d/event_handlers.rb' do source 'event_handlers.rb' action :nothing end.run_action(:create) We're dropping off our configuration file using cookbookf_file. Let's upload the cookbook and run the chef-client: [workstation] ec_handlers $ berks upload # after updating version ... [workstation] ec_handlers $ knife ssh 'name:web-node1' 'sudo chef-client' ... You'll notice that nothing printed out during this first run. That's because the chef-client configuration isn't reloaded during the run. If we run chef-client again then our handlers will run as we expect: [workstation] ec_handlers $ knife ssh 'name:web-node1' 'sudo chef-client' ... servername.mylabserver.com servername.mylabserver.com Starting Chef Client, version 13.10.4 servername.mylabserver.com servername.mylabserver.com Ohai completed!! servername.mylabserver.com servername.mylabserver.com Running JsonLogger with file: /var/log/chef-start.json servername.mylabserver.com resolving cookbooks for run list: ["ec_base", "ec_handlers", "ec_nginx::ohai", "ec_status_site::default"] servername.mylabserver.com Synchronizing Cookbooks: ... servername.mylabserver.com Recipe: ec_nginx::ohai servername.mylabserver.com * ohai_plugin[nginx] action create servername.mylabserver.com * directory[/etc/chef/ohai/plugins] action create (skipped due to not_if) servername.mylabserver.com Resource Skipped for web-node1 servername.mylabserver.com * cookbook_file[/etc/chef/ohai/plugins/nginx.rb] action create (up to date) servername.mylabserver.com * ohai[nginx] action nothing (skipped due to action :nothing) ... One last thing to note here is that the resource_skipped event apparently doesn't fire if the action is run in the compile phase, even if it is "up to date." Now we have a better idea of how we can configure Chef handlers even beyond the 3 base types (start, report, and exception) that Chef prescribes.

Hands-on Labs are real live environments that put you in a real scenario to practice what you have learned without any other extra charge or account to manage.

01:00:00

Definitions and Libraries

Libraries

What Are Libraries?

00:03:45

Lesson Description:

Resources are really the building blocks we use with Chef, but sometimes we need to use some extra Ruby logic to determine which resource to run for in a given situation. Often in those moments, we also find ourselves repeating the same bits of code in numerous locations. This isn't ideal, and libraries exist to help in these situations. Documentation For This Video Chef Libraries DocumentationPDF of Slideshow

Writing and Using Libraries - Part 1

00:09:11

Lesson Description:

Now that we have an understanding of what libraries are, we're going to look at how we can create them, and how they get loaded into our recipes. Documentation For This Video Chef Libraries DocumentationOfficial NGINX Linux Packages Expanding the ec_nginx Cookbook with Libraries Up to this point, we've been relying on the epel-release yum repository for installing NGINX, but that is less than ideal. We would like our cookbook to make it easy to install and setup NGINX on both Debian and Redhat systems. With that in mind, we're going to need to start conditionally running resources based on the system type, and this is a great time to extract out some library methods. From within our ec_nginx cookbook, let's generate a new library. Like usual, the chef CLI provides a generator for us, but this time it is named a little differently. To generate a new library we'll use the helpers generator. This command is odd because it is both plural and doesn't map directly to what we're trying to create. Let's create a library file called helpers for the time being: [workstation] ec_nginx $ chef generate helpers helpers Recipe: code_generator::helpers * directory[/home/user/chef-repo/cookbooks/ec_nginx/libraries] action create - create new directory /home/user/chef-repo/cookbooks/ec_nginx/libraries - restore selinux security context * template[/home/user/chef-repo/cookbooks/ec_nginx/libraries/helpers.rb] action create - create new file /home/user/chef-repo/cookbooks/ec_nginx/libraries/helpers.rb - update content in file /home/user/chef-repo/cookbooks/ec_nginx/libraries/helpers.rb from none to 7bfb3c (diff output suppressed by config) - restore selinux security context [workstation] ec_nginx $ Within this file, we actually get quite a bit of useful information about libraries: ~/chef-repo/cookbooks/ec_nginx/libraries/helpers.rb # # Chef Documentation # https://docs.chef.io/libraries.html # # # This module name was auto-generated from the cookbook name. This name is a # single word that starts with a capital letter and then continues to use # camel-casing throughout the remainder of the name. # module EcNginx module HelpersHelpers # # Define the methods that you would like to assist the work you do in recipes, # resources, or templates. # # def my_helper_method # # help method implementation # end end end # # The module you have defined may be extended within the recipe to grant the # recipe the helper methods you define. # # Within your recipe you would write: # # extend EcNginx::HelpersHelpers # # my_helper_method # # You may also add this to a single resource within a recipe: # # template '/etc/app.conf' do # extend EcNginx::HelpersHelpers # variables specific_key: my_helper_method # end # The documentation points us towards a more explicit use of the library by creating a module for us to put our methods into, and showing us how we can use Ruby's module mixin functionality to load those methods into our recipes. This isn't the only way to use libraries, and we'll explore how exactly libraries are loaded in a little later. For now, let's discuss what we need to do with our library to make it easy to setup NGINX on both Debian and Redhat systems. We need to do the following: Easily determine if the system is Debian or Redhat without needing to use node['platform_family'] all over our codeReference the proper file for the Official NGINX packages depending on the system Notice that neither of these goals is to download the file or install NGINX, those are the job of the resources that will leverage these library methods. Let's create these methods now: ~/chef-repo/cookbooks/ec_nginx/libraries/helpers.rb module EcNginx module Helpers def debian_based? node['platform_family'] == 'debian' end def rhel_based? node['platform_family'] == 'rhel' end def nginx_key_url 'http://nginx.org/keys/nginx_signing.key' end end end Notice that all of these methods are one line. Libraries don't have to be complicated, but by giving these values meaningful names we'll make the logic within our recipes easier to maintain and read in the future. We also changed the module name from HelpersHelpers to just Helpers so that it makes more sense. Writing Tests for Platform Support Now that we're trying to support more than one platform, we're ready to add Ubuntu to the matrix of systems that we run our integration tests against. Let's add a new platform to the .kitchen.yml: ~/chef-repo/cookbooks/ec_nginx/.kitchen.yml --- driver: name: docker use_sudo: false provisioner: name: chef_zero # You may wish to disable always updating cookbooks in CI or other testing environments. # For example: # always_update_cookbooks: <%= !ENV['CI'] %> always_update_cookbooks: true product_name: chef product_version: 13.10.4 verifier: name: inspec platforms: - name: ubuntu-16.04 driver_config: run_command: /bin/systemd privileged: true - name: centos-7 driver_config: run_command: /usr/sbin/init privileged: true provision_command: - systemctl enable sshd.service suites: - name: default run_list: - recipe[ec_nginx::default] - recipe[ec_nginx::ohai] verifier: inspec_tests: - test/integration/default attributes: We don't actually have any real integration tests just yet, so let's write some for our default recipe: ~/chef-repo/cookbooks/ec_nginx/test/integration/default/default_test.rb # # encoding: utf-8 # Inspec test for recipe ec_nginx::default # The Inspec reference, with examples and extensive documentation, can be # found at http://inspec.io/docs/reference/resources/ if os.redhat? describe file('/etc/yum.repos.d/nginx.repo') do it { should exist } end end describe package('nginx') do it { should be_installed } its('version') { should include '1.14.0' } end describe service('nginx') do it { should be_running } it { should be_enabled } end We've set the version based on what we know is the latest at the time of recording, you may need to adjust this in the future. We also need to adjust our Ohai tests so that they match: ~/chef-repo/cookbooks/ec_nginx/test/integration/default/ohai_test.rb describe command('ohai -d /tmp/kitchen/ohai/plugins nginx') do its(:stdout) { should include('nginx version: nginx/1.14.0') } end Now we've got some useful tests. Let's remove the bit about epel-release in our default recipe before running the tests for the first time: ~/chef-repo/cookbooks/ec_nginx/recipes/default.rb package 'nginx' service 'nginx' do action [:enable, :start] end Finally, let's run our tests: [workstation] ec_nginx $ kitchen test ... Profile: tests from {:path=>"/home/user/chef-repo/cookbooks/ec_nginx/test/integration/default"} (tests from {:path=>".home.user.chef-repo.cookbooks.$ c_nginx.test.integration.default"}) Version: (not specified) Target: ssh://kitchen@localhost:32770 Command ohai -d /tmp/kitchen/ohai/plugins nginx ? stdout should include "nginx version: nginx/1.12.2" expected "{n "version": "nginx version: nginx/1.10.3 (Ubuntu)\n"n}n" to include "nginx version: nginx/1.12.2" Diff: @@ -1,2 +1,4 @@ -nginx version: nginx/1.12.2 +{ + "version": "nginx version: nginx/1.10.3 (Ubuntu)n" +} System Package nginx ? should be installed ? version should eq "1.12.2" expected: "1.12.2" got: "1.10.3-0ubuntu0.16.04.2" (compared using ==) Service nginx ? should be running ? should be enabled Test Summary: 3 successful, 2 failures, 0 skipped ... Target: ssh://kitchen@localhost:32776 System Package nginx ? should be installed ? version should include "1.14.0" expected "1.10.3-0ubuntu0.16.04.2" to include "1.14.0" Service nginx ? should be running ? should be enabled Command ohai -d /tmp/kitchen/ohai/plugins nginx ? stdout should include "nginx version: nginx/1.14.0" expected "{n "version": "nginx version: nginx/1.10.3 (Ubuntu)\n"n}n" to include "nginx version: nginx/1.14.0" Diff: @@ -1,2 +1,4 @@ -nginx version: nginx/1.14.0 +{ + "version": "nginx version: nginx/1.10.3 (Ubuntu)n" +} Test Summary: 3 successful, 2 failures, 0 skipped ... Recipe: ec_nginx::default * yum_package[nginx] action install * No candidate version available for nginx ================================================================================ Error executing action `install` on resource 'yum_package[nginx]' ================================================================================ Chef::Exceptions::Package ------------------------- No candidate version available for nginx As we can see, on Ubuntu there is an NGINX package available by default, but it is pretty old. Without EPEL, there is no such package available for CentOS. This is a great starting point for us to learn how to utilize our library in the next lesson. Continued in Part 2

Writing and Using Libraries - Part 2

00:10:32

Lesson Description:

We've created our library and written some tests to help us verify that it works on various platforms, and now we're ready to learn how to use the library in our recipes and resources. Documentation For This Video Chef Libraries DocumentationOfficial NGINX Linux PackagesThe apt_repository ResourceThe yum_repository Resource Utilizing Our EcNginx::Helpers Module In the first part of this lecture, we left ourselves with some failing tests showing that we were no longer relying on the epel-release repository on Redhat based systems, and we were installing the wrong version of NGINX on Ubuntu. Now we're ready to use our helpers in our default recipe to make these tests pass: ~/chef-repo/cookbooks/ec_nginx/recipes/default.rb extend EcNginx::Helpers if debian_based? apt_repository 'nginx' do uri "http://nginx.org/packages/#{node['platform']}" components ['nginx'] key nginx_key_url end elsif rhel_based? yum_repository 'nginx' do baseurl "http://nginx.org/packages/#{node['platform']}/#{node['platform_version'][0]}/$basearch/" gpgkey nginx_key_url end end package 'nginx' service 'nginx' do action [:enable, :start] end Notice that we're using extend EcNginx::Helpers to essentially import the methods from within our module into the current context. From there, we're able to use the debian_based?, rhel_based?, and nginx_key_url methods. Depending on the platform, we're utilizing either the apt_repository resource or the yum_repository resource to add the official NGINX package to the list of available packages. Let's run kitchen test again to see what happens: [workstation] ec_nginx $ kitchen test ... ================================================================================ Recipe Compile Error in /tmp/kitchen/cache/cookbooks/ec_nginx/recipes/default.rb ================================================================================ NoMethodError ------------- undefined method `nginx_key_url' for Chef::Resource::YumRepository Cookbook Trace: --------------- /tmp/kitchen/cache/cookbooks/ec_nginx/recipes/default.rb:12:in `block in from_file' /tmp/kitchen/cache/cookbooks/ec_nginx/recipes/default.rb:10:in `from_file' ... We didn't get an error for our debian_based? or rhel_based? methods, but we did for the nginx_key_url. Notice that it says for Chef::Resource::YumRepository. Our extend EcNginx::Helpers line allowed us to pull them into the recipe, but not individual resources. To get around that we have a few different options: Place an extend EcNginx::Helpers line within the block for each of these resources.Have the base class for all resources, Chef::Resource, execute the line include EcNginx::Helpers The first option is very explicit, but a little tedious. The second option is a global solution and loads our helpers into every resource, except custom resources. Which of these you decide to do really depends on the nature of your helpers. Here's what our code would look like doing either approach: Utilizing extend: extend EcNginx::Helpers if debian_based? apt_repository 'nginx' do extend EcNginx::Helpers uri "http://nginx.org/packages/#{node['platform']}" components ['nginx'] key nginx_key_url end elsif rhel_based? yum_repository 'nginx' do extend EcNginx::Helpers baseurl "http://nginx.org/packages/#{node['platform']}/#{node['platform_version'][0]}/$basearch/" gpgkey nginx_key_url end end # Remainder of the recipe Mixing helpers into all resources: extend EcNginx::Helpers Chef::Resource.class_eval { include EcNginx::Helpers } if debian_based? apt_repository 'nginx' do uri "http://nginx.org/packages/#{node['platform']}" components ['nginx'] key nginx_key_url end elsif rhel_based? yum_repository 'nginx' do baseurl "http://nginx.org/packages/#{node['platform']}/#{node['platform_version'][0]}/$basearch/" gpgkey nginx_key_url end end If we really wanted to, we could utilize the Chef::Resource.class_eval { include EcNginx::Helpers } line from within the library itself so that it is available in all recipes. Additionally, if we were to run Chef::Recipe.class_eval { include EcNginx::Helpers } from within the library, it would automatically include our helpers in all recipes. For now, let's go with our latter example and re-run our tests: [workstation] ec_nginx $ kitchen test ... Target: ssh://kitchen@localhost:32778 System Package nginx ? should be installed ? version should include "1.14.0" Service nginx ? should be running ? should be enabled Command ohai -d /tmp/kitchen/ohai/plugins nginx ? stdout should include "nginx version: nginx/1.14.0" Test Summary: 5 successful, 0 failures, 0 skipped ... On both platforms, our tests are now passing. Using Libraries in Custom Resources The custom resource that we ship as part of the ec_nginx cookbook could obviously benefit from this enhancement to install the NGINX from the official repository. For this to work we need to ensure that it gets loaded into the class that is created for each action behind the scenes using the action_class method from within our custom resource: ~/chef-repo/cookbooks/ec_nginx/resources/server.rb resource_name :nginx_server property :host, String, name_property: true property :static_path, String property :landing_page_content, String property :cookbook, String, default: 'ec_nginx' action_class { include EcNginx::Helpers } action :create do if debian_based? apt_repository 'nginx' do extend EcNginx::Helpers uri "http://nginx.org/packages/#{node['platform']}" components ['nginx'] key nginx_key_url end elsif rhel_based? yum_repository 'nginx' do extend EcNginx::Helpers baseurl "http://nginx.org/packages/#{node['platform']}/#{node['platform_version'][0]}/$basearch/" gpgkey nginx_key_url end end package 'nginx' service 'nginx' do action [:start, :enable] end static_path = new_resource.static_path || "/var/www/#{new_resource.host}" directory static_path do recursive true end template "/etc/nginx/conf.d/#{new_resource.host}.conf" do source "nginx-server.conf.erb" cookbook new_resource.cookbook variables( host: new_resource.host, static_path: static_path ) notifies :reload, 'service[nginx]' end if new_resource.landing_page_content file "#{static_path}/index.html" do content new_resource.landing_page_content end end end action :delete do end With all of these changes, we should change our cookbook version and push the new version to the Chef Server, so that the next time a cookbook that relies on the nginx_server resource is used, it will use the proper NGINX package.

Sharing Libraries

00:03:38

Lesson Description:

The most common use for a library would be to use it within the cookbook that contains it, but occasionally we'll want to share libraries so other cookbook authors can use them. In this lesson, we'll look at how we can do that. Documentation For This Video Chef Libraries Documentation Sharing Libraries We've already seen that libraries can be used within custom resources that are being used by other cookbooks, and that might have hinted out how we share libraries: using cookbooks. The ec_status_site cookbook already relies on the ec_nginx cookbook so it will work as a playground for us to use the library. To see this in action, we'll add an execute resource to echo some information: ~/chef-repo/cookbooks/ec_status_site/recipes/default.rb extend EcNginx::Helpers if rhel_based? execute "display key information" do extend EcNginx::Helpers command "echo 'NGINX GPG key #{nginx_key_url}'" end end nginx_server 'status.example.com' do landing_page_content '<h1>Everything is Great!</h1>' end We still have to follow the same steps that we followed when using the library from within the ec_nginx cookbook. The methods from the module need to be loaded into both the recipe and the resources that need them. Now we'll upload our cookbook and run chef-client to see what happens: [workstation] ec_status_site $ berks upload --force ... [workstation] ec_status_site $ knife ssh 'name:web-node1' 'sudo chef-client' ... servername.mylabserver.com * execute[display key information] action run servername.mylabserver.com - execute echo 'NGINX GPG key http://nginx.org/keys/nginx_signing.key' ... By this point, we've gotten pretty used to the idea that cookbooks are how we package up most types of Chef code that we want to share, whether it's a recipe, resource, library, or an ohai hint.

Resources vs Libraries

00:14:37

Lesson Description:

For the most part, libraries are used to package up useful logic helpers, and custom resources manage groups of other resources. Occasionally, resources are created as libraries, and we'll look at when and why to do that in this lesson. Documentation For This Video Chef Custom Resources provides Documentation Creating Resources Using Libraries The way that we've created custom resources up to this point has been to put them in the /resources directory and to use the custom resource DSL. This is great, and is the recommended way to do it. But if you're creating a resource that works on numerous platforms, sometimes it is nice to avoid using the custom resource DSL and to instead use a library. Continuing to work with our nginx_server resource, let's see what this would look like if we had created it as a library instead. Before we begin, let's use git to commit the cookbooks as they exist now so that we can come back to this point after we've experimented with library based resources: [workstation] ec_nginx $ rm -rf .delivery [workstation] ec_nginx $ git init [workstation] ec_nginx $ git add --all . [workstation] ec_nginx $ git commit -m 'nginx_server using custom resource DSL' 21 files changed, 455 insertions(+) create mode 100644 .gitignore create mode 100644 .kitchen.yml create mode 100644 Berksfile create mode 100644 LICENSE create mode 100644 README.md create mode 100644 chefignore create mode 100644 files/default/nginx.rb create mode 100644 libraries/helpers.rb create mode 100644 metadata.rb create mode 100644 recipes/default.rb create mode 100644 recipes/ohai.rb create mode 100644 resources/server.rb create mode 100644 spec/spec_helper.rb create mode 100644 spec/unit/recipes/default_spec.rb create mode 100644 spec/unit/recipes/ohai_spec.rb create mode 100644 spec/unit/resources/server_spec.rb create mode 100644 templates/nginx-server.conf.erb create mode 100644 test/fixtures/cookbooks/nginx_test/metadata.rb create mode 100644 test/fixtures/cookbooks/nginx_test/recipes/default.rb create mode 100644 test/integration/default/default_test.rb create mode 100644 test/integration/default/ohai_test.rb Now we can create a new branch to experiment with: [workstation] ec_nginx $ git checkout -b library-based-resources Switched to a new branch 'library-based-resources' From here, let's create a subdirectory in the libraries directory called nginx: [workstation] ec_nginx $ mkdir -p libraries/nginx Our first step will be to create a server.rb that behaves exactly like what we have right now by creating a module and a Chef::Resource subclass: ~/chef-repo/cookbooks/ec_nginx/libraries/nginx/server.rb module EcNginx class NginxServer < Chef::Resource # resource code goes here end end This is really what we get for free when we create our custom resources within the resources/ directory. Now we can copy all of the contents from the resources/server.rb file and place them within this class: ~/chef-repo/cookbooks/ec_nginx/libraries/nginx/server.rb module EcNginx class NginxServer < Chef::Resource resource_name :nginx_server property :host, String, name_property: true property :static_path, String property :landing_page_content, String property :cookbook, String, default: 'ec_nginx' action_class { include EcNginx::Helpers } action :create do if debian_based? apt_repository 'nginx' do extend EcNginx::Helpers uri "http://nginx.org/packages/#{node['platform']}" components ['nginx'] key nginx_key_url end elsif rhel_based? yum_repository 'nginx' do extend EcNginx::Helpers baseurl "http://nginx.org/packages/#{node['platform']}/#{node['platform_version'][0]}/$basearch/" gpgkey nginx_key_url end end package 'nginx' service 'nginx' do action [:start, :enable] end static_path = new_resource.static_path || "/var/www/#{new_resource.host}" directory static_path do recursive true end template "/etc/nginx/conf.d/#{new_resource.host}.conf" do source "nginx-server.conf.erb" cookbook new_resource.cookbook variables( host: new_resource.host, static_path: static_path ) notifies :reload, 'service[nginx]' end if new_resource.landing_page_content file "#{static_path}/index.html" do content new_resource.landing_page_content end end end action :delete do end end end Next, we'll remove the resources directory and run our ChefSpec tests again: [workstation] ec_nginx $ rm -rf resources [workstation] ec_nginx $ chef exec rspec ........ Finished in 1.46 seconds (files took 1.83 seconds to load) 8 examples, 0 failures The library is loaded and the resource is registered properly. When to Use Library Resources As we can see, we didn't really gain a whole lot by doing this, but one of the benefits is that we can now separate out the implementation based on the platform. In our case, this is debian vs rhel. Let's create a new file at libraries/nginx/server_debian.rb: ~/chef-repo/cookbooks/ec_nginx/libraries/nginx/server_debian.rb module EcNginx class NginxServerDebian < NginxServer resource_name :nginx_server provides :nginx_server, platform_family: 'debian' action :create do apt_repository 'nginx' do extend EcNginx::Helpers uri "http://nginx.org/packages/#{node['platform']}" components ['nginx'] key nginx_key_url end super() end end end We'll do the same for our rhel implementation: ~/chef-repo/cookbooks/ec_nginx/libraries/nginx/server_rhel.rb module EcNginx class NginxServerRhel < NginxServer resource_name :nginx_server provides :nginx_server, platform_family: 'rhel' action :create do yum_repository 'nginx' do extend EcNginx::Helpers baseurl "http://nginx.org/packages/#{node['platform']}/#{node['platform_version'][0]}/$basearch/" gpgkey nginx_key_url end super() end end end Note: the name of these files is important because the server.rb file must be loaded first so that the NginxServer class can be subclasses. The files are then loaded in alphabetical order. If we had used the filenames of default.rb, debian.rb, and rhel.rb, then the debian.rb would be loaded first and raise an exception. When we call super() in the action :create block, that will execute the code that exists in the parent class's implementation of action :create. This allows us to separate out our platform-specific installation code and keep the same shared website setup code. Now we're able to remove the if ... elsif statement from the libraries/nginx/server.rb file. ~/chef-repo/cookbooks/ec_nginx/libraries/nginx/server.rb module EcNginx class NginxServer < Chef::Resource resource_name :nginx_server property :host, String, name_property: true property :static_path, String property :landing_page_content, String property :cookbook, String, default: 'ec_nginx' action_class { include EcNginx::Helpers } action :create do package 'nginx' service 'nginx' do action [:start, :enable] end static_path = new_resource.static_path || "/var/www/#{new_resource.host}" directory static_path do recursive true end template "/etc/nginx/conf.d/#{new_resource.host}.conf" do source "nginx-server.conf.erb" cookbook new_resource.cookbook variables( host: new_resource.host, static_path: static_path ) notifies :reload, 'service[nginx]' end if new_resource.landing_page_content file "#{static_path}/index.html" do content new_resource.landing_page_content end end end action :delete do end end end Utilizing nginx_test::default For InSpec Currently, we don't have any automated tests that exercise this code, so we're going to separate out some of our existing code to work with another test suite in our .kitchen.yml: ~/chef-repo/cookbooks/ec_nginx/.kitchen.yml # top of file omitted suites: - name: default run_list: - recipe[ec_nginx::default] - recipe[ec_nginx::ohai] verifier: inspec_tests: - test/integration/default - test/integration/ohai attributes: - name: nginx_server run_list: - recipe[nginx_test::default] verifier: inspec_tests: - test/integration/default The big change here is that we added a new suite called nginx_server. We've also pointed out that we should separate the ohai_test.rb into a different directory so that it won't run by default in our nginx_server suite. Let's create that directory and move the file now: [workstation] ec_nginx $ mkdir test/integration/ohai [workstation] ec_nginx $ mv test/integration/{default,ohai}/ohai_test.rb Finally, if we run kitchen test, it will run both suites using four containers. For right now, let's just run our nginx_server suite specifically: [workstation] ec_nginx $ kitchen test nginx-server ... Target: ssh://kitchen@localhost:32774 System Package nginx ? should be installed ? version should include "1.14.0" Service nginx ? should be running ? should be enabled Test Summary: 4 successful, 0 failures, 0 skipped ... Target: ssh://kitchen@localhost:32775 File /etc/yum.repos.d/nginx.repo ? should exist System Package nginx ? should be installed ? version should include "1.14.0" Service nginx ? should be running ? should be enabled Test Summary: 5 successful, 0 failures, 0 skipped ...

Hands-on Labs are real live environments that put you in a real scenario to practice what you have learned without any other extra charge or account to manage.

02:00:00

Definitions

What are Definitions?

00:03:10

Lesson Description:

Definitions are a subject that you need to know about for the Extending Chef Badge exam, but they are a feature you shouldn't really use in modern Chef. Documentation For This Video Chef Definitions DocumentationPDF of Slideshow

Knife Plugins

Knife Source Code

What are Knife Plugins?

00:03:21

Lesson Description:

Knife is one of the most-used tools in the Chef toolbox, and knife plugins allow us to add even more functionality to this fantastic tool. Documentation For This Video Knife Cloud PluginsCustom Knife PluginsCommunity Knife PluginsKnife common optionsPDF of Slideshow

Creating a Knife Plugin - Part 1

00:13:14

Lesson Description:

If we find ourselves in a position where Knife doesn't quite provide us with the functionality that we need, then the next course of action is to write a custom Knife plugin. In this lesson, we'll create our first Knife plugin. Documentation For This Video Knife Plugin DocumentationCustom Knife PluginsKnife Common OptionsCommunity Knife PluginsThe terminal-table GemThe terminal-table GitHub Our Example Knife Plugin Unlike most of the extensions that we've been making for Chef, Knife plugins are not distributed using cookbooks. This is because Knife plugins are run from workstations instead of Chef nodes. Because of this, we're able to (a) install other Knife plugins using the chef gem command, (b) package knife commands as part of our chef-repo, and (c) develop custom plugins locally with ease. To learn more about Knife plugins, we're going to create one that interacts with the Chef Server to present some information to us about our nodes in a nice table. By the time we're finished, we'll be able to type the following command and see similar results: [workstation] chef-repo $ knife node table +-----------+---------------------------+ | Name | Last Check-In | +-----------+---------------------------+ | web-node1 | 2018-09-19 19:33:20 +0000 | +-----------+---------------------------+ To start, let's create a new directory within our chef-repo to hold onto Knife plugins, and then let's create a new plugin file called node_table.rb [workstation] chef-repo $ mkdir -p .chef/plugins/knife [workstation] chef-repo $ touch .chef/plugins/knife/node_table.rb Now we're ready to start developing our Knife plugin. Creating the Plugin Skeleton We need a class that will register itself as a Knife plugin, so our first step will be to create one that inherits from Chef::Knife. It is also a good practice to wrap the class within a module to create a namespace. Let's create the simplest class we can for now: ~/chef-repo/.chef/plugins/knife/node_table.rb module ExampleCom class NodeTable < Chef::Knife def run puts "Hello, World!" end end end Now let's see if it works: [workstation] chef-repo $ knife node table Hello, World! Success! This worked because we named the class NodeTable. Knife will split this so that each uppercased character begins another word in the subcommand, so NodeTable becomes node table. Unfortunately, if we look under the knife node command we don't currently see anything about node table: [workstation] chef-repo $ knife node --help FATAL: Cannot find subcommand for: 'node --help' Available node subcommands: (for details, knife SUB-COMMAND --help) ** NODE COMMANDS ** knife node bulk delete REGEX (options) knife node create NODE (options) knife node delete [NODE [NODE]] (options) knife node edit NODE (options) knife node environment set NODE ENVIRONMENT knife node from file FILE (options) knife node list (options) knife node run_list add [NODE] [ENTRY [ENTRY]] (options) knife node run_list remove [NODE] [ENTRY [ENTRY]] (options) knife node run_list set NODE ENTRIES (options) knife node show NODE (options) knife node status [<node> <node> ...] Usage: /opt/chefdk/bin/knife (options) Thankfully, we can add our command to this list using the banner method that the Chef::Knife class provides us with. ~/chef-repo/.chef/plugins/knife/node_table.rb module ExampleCom class NodeTable < Chef::Knife banner "knife node table [SEARCH] (options)" def run puts "Hello, World!" end end end Now if we look at the subcommands of knife node we should see our command: [workstation] chef-repo $ knife node --help FATAL: Cannot find subcommand for: 'node --help' Available node subcommands: (for details, knife SUB-COMMAND --help) ** NODE COMMANDS ** knife node bulk delete REGEX (options) knife node create NODE (options) knife node delete [NODE [NODE]] (options) knife node edit NODE (options) knife node environment set NODE ENVIRONMENT knife node from file FILE (options) knife node list (options) knife node run_list add [NODE] [ENTRY [ENTRY]] (options) knife node run_list remove [NODE] [ENTRY [ENTRY]] (options) knife node run_list set NODE ENTRIES (options) knife node show NODE (options) knife node status [<node> <node> ...] knife node table [SEARCH] (options) Now we're ready to start working on the actual functionality. Handling User Input The first thing that we want the user to be able to do is provide a SEARCH query to filter the results down. By default we also want the query to return all of the nodes. Let's add some code to show how we can read in an optional positional argument: ~/chef-repo/.chef/plugins/knife/node_table.rb module ExampleCom class NodeTable < Chef::Knife banner "knife node table [SEARCH] (options)" def run search = name_args.first || "name:*" puts "SEARCH value is #{search}" end end end The name_args method is going to be how we read in positional values. Let's see if this worked: [workstation] chef-repo $ knife node table SEARCH value is name:* [workstation] chef-repo $ knife node table name:web-* SEARCH value is name:web-* Now that we can receive a custom query string, we should move onto implementing the search. Relying on Chef functionality and Knife commands When our plugin is loaded, we are able to trigger lazy loading of other dependencies using the deps method. In our case, we're going to want to load in the Chef::Search::Query class that can be found at the gem path of chef/search/query. Let's load this in now and make our preliminary search: ~/chef-repo/.chef/plugins/knife/node_table.rb module ExampleCom class NodeTable < Chef::Knife banner "knife node table [SEARCH] (options)" deps do require 'chef/search/query' end def run query = name_args.first || "name:*" Chef::Search::Query.new.search('node', query, filter_result: { 'ohai_time' => ['ohai_time'], 'name' => ['name'], }) do |node| checked_in = node['ohai_time'] ? Time.at(node['ohai_time']) : nil ui.msg("Node: #{node['name']} converged at #{checked_in}") end end end end We're utilizing deps to load in the code we need from Chef, and once we've made our query we're utilizing the ui object to display the information from the node (for now). There's a lot more we can do with the ui object, so take time to read the documentation on it. The last check-in time for a node is stored in the ohai_time attribute as a float. Before we can render it as a moment in time, we need to convert it to a Time object using the Time.at method. Running this again without a query we now get: [workstation] chef-repo $ knife node table Node: web-node1 converged at 2018-09-19 19:33:20 +0000 Now we have the information that we need, and that's left to do is to format it into a table. Utilizing Third-Party Gems To handle the table formatting we're going to pull in the terminal-table gem. Let's put the require statement in our deps and utilize the gem: ~/chef-repo/.chef/plugins/knife/node_table.rb module ExampleCom class NodeTable < Chef::Knife banner "knife node table [SEARCH] (options)" deps do require 'chef/search/query' require 'terminal-table' end def run query = name_args.first || "name:*" table = Terminal::Table.new(headings: ['Name', 'Last Check-In']) Chef::Search::Query.new.search('node', query, filter_result: { 'ohai_time' => ['ohai_time'], 'name' => ['name'], }) do |node| checked_in = node['ohai_time'] ? Time.at(node['ohai_time']) : nil table.add_row([node['name'], checked_in]) end ui.msg(table) end end end We're creating a Terminal::Table object that's provided by the gem, adding a row per node in our search results, and finally using ui.msg to display the table. Let's see it in action: [workstation] chef-repo $ knife node table /opt/chefdk/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:59:in `require': cannot load such file -- terminal-table (LoadError) from /opt/chefdk/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:59:in `require' from /home/user/chef-repo/.chef/plugins/knife/node_table.rb:6:in `block in <class:NodeTable>' from /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/chef-13.10.4/lib/chef/knife.rb:233:in `block in load_deps' from /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/chef-13.10.4/lib/chef/knife.rb:232:in `each' from /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/chef-13.10.4/lib/chef/knife.rb:232:in `load_deps' from /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/chef-13.10.4/lib/chef/knife.rb:217:in `run' from /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/chef-13.10.4/lib/chef/application/knife.rb:160:in `run' from /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/chef-13.10.4/bin/knife:25:in `<top (required)>' from /opt/chefdk/bin/knife:267:in `load' from /opt/chefdk/bin/knife:267:in `<main>' We're running into an error because we never installed the gem. Since we need this gem for our Knife plugin to work, it needs to be installed into the same location as Knife itself. And for that we should always use chef gem install ... when working with the ChefDK. Let's add the gem now and try again: [workstation] chef-repo $ chef gem install terminal-table Fetching: terminal-table-1.8.0.gem (100%) Successfully installed terminal-table-1.8.0 1 gem installed [workstation] chef-repo $ knife node table +-----------+---------------------------+ | Name | Last Check-In | +-----------+---------------------------+ | web-node1 | 2018-09-19 19:33:20 +0000 | +-----------+---------------------------+ It works! Let's see what our edge cases look like before calling this finished though. What happens when we: Use a valid search that yields no results?Use an invalid search? [workstation] chef-repo $ knife node table name:fake +------+---------------+ | Name | Last Check-In | +------+---------------+ +------+---------------+ [workstation] chef-repo $ knife node table fasdasdfadf:fasdfasdf:asdfasdfasdf ERROR: The data in your request was invalid Response: invalid search query: 'fasdasdfadf:fasdfasdf:asdfasdfasdf' It looks like we'll need to handle errors with our query first and return an error message if there is one. If there are no results, we should print out a useful message instead of the table. Here's one way to accomplish this: ~/chef-repo/.chef/plugins/knife/node_table.rb module ExampleCom class NodeTable < Chef::Knife banner "knife node table [SEARCH] (options)" deps do require 'chef/search/query' require 'terminal-table' end def run query = name_args.first || "name:*" table = Terminal::Table.new(headings: ['Name', 'Last Check-In']) begin Chef::Search::Query.new.search('node', query, filter_result: { 'ohai_time' => ['ohai_time'], 'name' => ['name'], }) do |node| checked_in = node['ohai_time'] ? Time.at(node['ohai_time']) : nil table.add_row([node['name'], checked_in]) end rescue ui.error("Invalid search query: #{query}") exit 1 end table.rows.length > 0 ? ui.msg(table) : ui.msg("Search yielded zero nodes.") end end end Finally, let's try our edge cases one more time. [workstation] chef-repo $ knife node table fasdasdfadf:fasdfasdf:asdfasdfasdf ERROR: Invalid search query: fasdasdfadf:fasdfasdf:asdfasdfasdf [workstation] chef-repo $ knife node table name:fake Search yielded zero nodes. [workstation] chef-repo $ knife node table name:web* +-----------+---------------------------+ | Name | Last Check-In | +-----------+---------------------------+ | web-node1 | 2018-09-19 19:33:20 +0000 | +-----------+---------------------------+ Note: Continued in Part 2.

Creating a Knife Plugin - Part 2

00:11:29

Lesson Description:

This is a continuation from Part 1. Documentation For This Video Knife Plugin DocumentationCustom Knife PluginsKnife Common OptionsCommunity Knife PluginsThe terminal-table GemThe terminal-table GitHub Adding Options Currently, we have an optional argument in the form of SEARCH, but what if we wanted to have more options that could be passed in based on a flag? For that, the Chef::Knife class provides the option method to allow us to define them. Let's add an option to our plugin that will allow us to specify whether there should or should not be a header: ~/chef-repo/.chef/plugins/knife/node_table.rb module ExampleCom class NodeTable < Chef::Knife banner "knife node table [SEARCH] (options)" deps do require 'json' require 'chef/search/query' require 'terminal-table' end option :headers, long: '--[no-]headers', description: 'Set display of table headers', boolean: true, default: true option :style, short: '-s JSON', long: '--style JSON', description: 'JSN String to set styles for terminal-table', proc: Proc.new { |j| JSON.parse(j, symbolize_keys: true) }, default: {} def run query = name_args.first || "name:*" if config[:headers] table = Terminal::Table.new(headings: ['Name', 'Last Check-In']) else table = Terminal::Table.new end table.style = config[:style] begin Chef::Search::Query.new.search('node', query, filter_result: { 'ohai_time' => ['ohai_time'], 'name' => ['name'], }) do |node| checked_in = node['ohai_time'] ? Time.at(node['ohai_time']) : nil table.add_row([node['name'], checked_in]) end rescue ui.error("Invalid search query: #{query}") exit 1 end table.rows.length > 0 ? ui.msg(table) : ui.msg("Search yielded zero nodes.") end end end With these two option methods, we've demonstrated most of the functionality present for this method. When creating a boolean switch, Knife will consider whether the flag includes the word no, and set the value to false. We're able to read off the values of these options using the config object. The style option is more interesting. We're expecting it to be a Hash with symbol keys by the time it reaches the run method. To ensure that this happens, we've parsed the value passed in using the proc setting. This Proc will be processed whenever this option is present and it happens before we make it to the run method. Let's take a look at some different output combinations that we were able to achieve using these options: [workstation] chef-repo $ knife node table +-----------+---------------------------+ | Name | Last Check-In | +-----------+---------------------------+ | web-node1 | 2018-09-19 19:33:20 +0000 | | example | | +-----------+---------------------------+ [workstation] chef-repo $ knife node table --no-header -s '{"border_bottom": false,"border_x": "="}' +===========+===========================+ | web-node1 | 2018-09-19 19:33:20 +0000 | | example | | [workstation] chef-repo $ Now we have a useful Knife plugin, and we're ready to see how to package this up for others to use.

Packaging and Sharing a Knife Plugin

00:12:57

Lesson Description:

Since Knife plugins are used outside the context of the Chef Client run, we share them as Ruby gems. In this lesson, we'll take the knife node table plugin that we created in the previous lesson and package it up as a Ruby gem to share. Documentation For This Video Bundler DocumentationThe bundle gem CommandRubyGems.orgRubyGems Publishing Guide Creating a Ruby Gem The main way that Ruby gems are created is by using the bundle gem command provided by Bundler. This will generate a gem skeleton that follows the best practices used by the Ruby community when creating gems. As a best practice, we will prefix our gem name with knife- to indicate that it is a Knife plugin. Let's move out of our chef-repo and create our gem as knife-node-table: [workstation] chef-repo $ cd ~ [workstation] ~ $ chef exec bundle gem knife-node-table Creating gem 'knife-node-table'... Do you want to generate tests with your gem? Type 'rspec' or 'minitest' to generate those test files now and in the future. rspec/minitest/(none): Do you want to license your code permissively under the MIT license? This means that any other developer or company will be legally allowed to use your code for free as long as they admit you created it. You can read more about the MIT license at http://choosealicense.com/licenses/mit. y/(n): Do you want to include a code of conduct in gems you generate? Codes of conduct can increase contributions to your project by contributors who prefer collaborative, safe spaces. You can read more about the code of conduct at contributor-covenant.org. Having a code of conduct means agreeing to the responsibility of enforcing it, so be sure that you are prepared to do that. Be sure that your email address is specified as a contact in the generated code of conduct so that people know who to contact in case of a violation. For suggestions about how to enforce codes of conduct, see http://bit.ly/coc-enforcement. y/(n): create knife-node-table/Gemfile create knife-node-table/lib/knife/node/table.rb create knife-node-table/lib/knife/node/table/version.rb create knife-node-table/knife-node-table.gemspec create knife-node-table/Rakefile create knife-node-table/README.md create knife-node-table/bin/console create knife-node-table/bin/setup create knife-node-table/.gitignore Initializing git repo in /home/user/knife-node-table This is fairly close, but we're going to move some of the files around because we don't quite want the knife/node/table structure. We really wanted the Rakefile, knife-node-table.gemspec, and other support files. Let's make some changes to our structure and move over our source file: [workstation] ~ $ cd knife-node-table [workstation] knife-node-table $ mkdir lib/knife-node-table [workstation] knife-node-table $ mv lib/{knife/node/table,knife-node-table}/version.rb [workstation] knife-node-table $ rm -rf lib/knife [workstation] knife-node-table $ mkdir -p lib/chef/knife [workstation] knife-node-table $ mv ~/chef-repo/.chef/plugins/knife/node_table.rb lib/chef/knife/ Finally, let's wrap our NodeTable class to be within the Chef::Knife namespace: ~/knife-node-table/lib/chef/knife/node_table.rb class Chef class Knife class NodeTable < Knife banner "knife node table [SEARCH] (options)" deps do require 'json' require 'chef/search/query' require 'terminal-table' end option :headers, long: '--[no-]headers', description: 'Set display of table headers', boolean: true, default: true option :style, short: '-s JSON', long: '--style JSON', description: 'JSN String to set styles for terminal-table', proc: Proc.new { |j| JSON.parse(j, symbolize_keys: true) }, default: {} def run query = name_args.first || "name:*" if config[:headers] table = Terminal::Table.new(headings: ['Name', 'Last Check-In']) else table = Terminal::Table.new end table.style = config[:style] begin Chef::Search::Query.new.search('node', query, filter_result: { 'ohai_time' => ['ohai_time'], 'name' => ['name'], }) do |node| checked_in = node['ohai_time'] ? Time.at(node['ohai_time']) : nil table.add_row([node['name'], checked_in]) end rescue ui.error("Invalid search query: #{query}") exit 1 end table.rows.length > 0 ? ui.msg(table) : ui.msg("Search yielded zero nodes.") end end end end We only changed the path and class name so that it follows Knife plugin standards, and can be used by other Knife plugins by using require 'chef/knife/node_table' within there dependencies. We're not quite done yet. We also need to make a modification to the knife-node-table.gemspec so that it loads the Knife::Node::Table class from the proper file and includes our dependency on terminal-table: ~/knife-node-table/knife-node-table.gemspec lib = File.expand_path("../lib", __FILE__) $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib) require "knife-node-table/version" Gem::Specification.new do |spec| spec.name = "knife-node-table" spec.version = Knife::Node::Table::VERSION spec.authors = ["Author Name"] spec.email = ["author@email.com"] spec.summary = %q{Adds `knife node table` subcommand.} spec.description = %q{Render a table with nodes' last known check-in time using `knife node table`.} spec.homepage = "" # Prevent pushing this gem to RubyGems.org. To allow pushes either set the 'allowed_push_host' # to allow pushing to a single host or delete this section to allow pushing to any host. if spec.respond_to?(:metadata) spec.metadata["allowed_push_host"] = "TODO: Set to 'http://mygemserver.com'" else raise "RubyGems 2.0 or newer is required to protect against " "public gem pushes." end spec.files = `git ls-files -z`.split("x0").reject do |f| f.match(%r{^(test|spec|features)/}) end spec.bindir = "exe" spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) } spec.require_paths = ["lib"] spec.add_dependency "terminal-table", "~> 1.8.0" spec.add_development_dependency "bundler", "~> 1.16" spec.add_development_dependency "rake", "~> 10.0" end The lines to pay attention to here are the require "knife-node-table/version" line and the spec.add_dependency... line. We also needed to remove TODO from the descriptions so that Bundler will build our gem later. With everything moved around, we're now ready to see our gem in action, but let's first make sure that it doesn't work right now: [workstation] knife-node-table $ chef gem uninstall terminal-table Successfully uninstalled terminal-table-1.8.0 [workstation] knife-node-table $ cd ~/chef-repo [workstation] chef-repo $ rm -rf .chef/plugins [workstation] chef-repo $ knife node table FATAL: Cannot find subcommand for: 'node table' Available node subcommands: (for details, knife SUB-COMMAND --help) ** NODE COMMANDS ** knife node bulk delete REGEX (options) knife node create NODE (options) knife node delete [NODE [NODE]] (options) knife node edit NODE (options) knife node environment set NODE ENVIRONMENT knife node from file FILE (options) knife node list (options) knife node run_list add [NODE] [ENTRY [ENTRY]] (options) knife node run_list remove [NODE] [ENTRY [ENTRY]] (options) knife node run_list set NODE ENTRIES (options) knife node show NODE (options) knife node status [<node> <node> ...] We've uninstalled the terminal-table gem manually so that we can make sure it will be installed as a dependency of our own gem, and Knife isn't currently picking up our plugin at all. Now we're ready to install our gem. Installing the Plugin as a Gem Usually, gems are installed from the RubyGem.org, but that is not the only way to install them. If you have a private gem server running, then you can add it to the gem command's source list and install gems from there. That's a bit beyond the scope of what we're doing here though. You can also install a gem using its source code, by manually building the corresponding .gem file for it and then using gem install with that file. That's what we're going to do. Thankfully the gem skeleton that we have gave us a handy command to generate this package. Before we build our gem we will need to commit what we currently have, so that the spec file will add the existing files to the gem. It would try to add files we deleted otherwise. Here's what the whole process looks like: [workstation] chef-repo $ cd ~/knife-node-table [workstation] knife-node-table $ git add --all . [workstation] knife-node-table $ git commit -m 'Initial commit' [master (root-commit) 01b6ea2] Initial commit 11 files changed, 171 insertions(+) create mode 100644 .gitignore create mode 100644 Gemfile create mode 100644 Gemfile.lock create mode 100644 README.md create mode 100644 Rakefile create mode 100755 bin/console create mode 100755 bin/setup create mode 100644 knife-node-table.gemspec create mode 100644 knife-node-table.gemspec.lock create mode 100644 lib/chef/knife/node_table.rb create mode 100644 lib/knife-node-table/version.rb [workstation] knife-node-table $ chef exec rake build knife-node-table 0.1.0 built to pkg/knife-node-table-0.1.0.gem. [workstation] knife-node-table $ chef gem install pkg/knife-node-table-0.1.0.gem Fetching: terminal-table-1.8.0.gem (100%) Successfully installed terminal-table-1.8.0 Successfully installed knife-node-table-0.1.0 2 gems installed We can see that it properly installed the terminal-table dependency in addition to just installing our gem. The last thing that we need to do is to make sure that Knife now has the subcommand. [workstation] knife-node-table $ knife node table +-----------+---------------------------+ | Name | Last Check-In | +-----------+---------------------------+ | web-node1 | 2018-09-19 19:33:20 +0000 | +-----------+---------------------------+ Sharing This Plugin If you create a Knife plugin for use within your organization, where should you store it? If it's proprietary, then you'll want to consider also deploying a Ruby gem server. But if you open-source the plugin, you can publish it to RubyGems.org. We didn't publish our gem in this course, to spare the public community from having a ton of duplicate gems (with everybody having to create one with a slightly different name). You can learn more about the gem publishing process by looking at the RubyGems.org publishing guide, but this won't be on the exam.

Common Knife Plugins and Private Clouds

00:03:27

Lesson Description:

Now that we know how to create and share a Knife plugin we're ready to look at some existing plugins. While exploring the plugin landscape we'll also discuss when it's a good choice to create a custom Knife plugin. Documentation For This Video Knife Custom PluginsKnife Community PluginsKnife Cloud Plugins The Most Common Type of Plugin: Cloud Plugins When it comes to types of Knife plugins, there is none more common than the cloud plugin. Chef themselves have created a number of cloud plugins that can be used to interact with the APIs of various public cloud providers. For the sake of the exam, we don't need to know how to use these exactly. But it's still good to know that they exist. They can add some great commands to make provisioning, bootstrapping, and managing public cloud infrastructure very easy. One of the most common reasons to create a custom Knife plugin is for encapsulating interactions that you need to have with a private cloud. If you work with an on-premises data center, then it may be useful to create your own custom Knife plugin to wrap the common actions that you and your team take when interacting with your own infrastructure. Other Community Plugins There are quite a few other plugins created by the community that add functionality to Knife and aren't related to a public or private cloud. A great place to look for these plugins is on the Community Plugins page in the Chef documentation. Another option would be to search RubyGems.org for gems that begin with knife- since that is the convention when naming Knife plugins.

Hands-on Labs are real live environments that put you in a real scenario to practice what you have learned without any other extra charge or account to manage.

01:30:00

Chef API

Using the Chef Server API

Communicating with the Chef API

00:09:43

Lesson Description:

Working from the command line to communicate with the Chef Server is very nice with knife, but occasionally we want other machines or software (say, a different web application) to interact with the server. For this, we can use the Chef Server REST API. In this lesson, we'll look at how to make a request to the Chef Server API using a language other than Ruby. Documentation For This Video Chef Server API DocumentationChef Server API Required HeadersChef Server API Authentication Documentation Chef Server API Request Requirements Before we look at the requirements for connecting to the Chef Server API, we should acknowledge that it's possible to make requests to the Chef Server API using any programming language capable of making an HTTP request. That being said, it's more cumbersome in some languages (like bash) than it is in others that have libraries to make the connection easier. For every request, we need some very specific headers, and some of them need to be hashed and Base64 encoded. The added layer of hashing and encoding makes it a little tedious to build up manually. This course doesn't require any prior knowledge of programming languages, other than Ruby, so we're going to survey the various ways that we can make requests by digging through the documentation. Examining a cURL Request to the Chef Server On the authentication documentation page, there's an example of using cURL to make a request. It's pretty complicated, but shows what would need to be done in any other language. Organization Specific Requests Vs /organizations For the most part, we need to make requests to paths that include an organization's name, such as /organizations/linuxacademy/nodes. This is an example of an organization-specific endpoint. There is a top-level /organizations endpoint, but it can only be accessed by the Chef Server's pivotal user (the user created when the Chef Server was created). If we need to make a request using the pivotal user, then we'll need to copy the pivotal.pem from the Chef Server. Let's see where that is by logging into our Chef Server and running a search: [chef-server] $ sudo find / -name 'pivotal.pem' /etc/opscode/pivotal.pem Chef, the company, used to be called Opscode. Occasionally there are files and directories that include this, so don't let that confuse us. Don't Always Trust the Documentation The Chef documentation is really quite nice, but that doesn't mean it is always accurate. One example is that there is no longer a Chef::REST class as used in an example in the API documentation. Instead, the class that should be used for talking to the Chef Server REST API from Ruby is now called Chef::ServerAPI. Keep these things in mind as you continue learning Chef, and be sure to always experiment with what you're reading in the documentation.

Hands-on Labs are real live environments that put you in a real scenario to practice what you have learned without any other extra charge or account to manage.

01:30:00

Course Conclusion

Final Steps

How to Prepare for the Exam

00:02:25

Lesson Description:

IMPORTANT: Don't forget to use the code LINUXACADEMY10 to get a discount when registering for the exam. You've learned everything that you need to know to take the exam and attain your Extending Chef Badge. To prepare for the exam I encourage you to do the following: Deploy your own Chef Server and a few Nodes if you didn't while following along with the course.Go through all of this course's Learning Activities.Complete the practice exam until you feel very comfortable with the content (this includes the hands-on portion).Read over the Extending Chef Badge scope document.Utilize the search functionality of docs.chef.io to ensure that you can quickly look up information that you don't know.Ensure that you understand how to write and deploy: Ohai PluginsChef HandlersCustom ResourcesLibrariesKnife Plugins Be aware of the debugging techniques that you have at your disposal, such as using Pry breakpoints, irb, and the chef-shell.Be familiar with why and how you could utilize the Chef Server REST API.Be familiar with the antiquated parts of Chef that you will be asked about: Definitionsuse_inline_resources The exam is mostly multiple choice, but there will be a few tasks that require you to utilize the workstation that you've been given to complete a task.

Hands-on Labs are real live environments that put you in a real scenario to practice what you have learned without any other extra charge or account to manage.

00:30:00

Extending Chef Badge Practice Exam

01:00:00

What's Next?

What's Next After Certification?

00:01:43

Lesson Description:

If you've successfully taken and passed the Extending Chef Badge exam then congratulations! If you haven't take the exam yet, you should feel confident going into the exam if you've diligently followed along with the examples in this course, taken practice exam, and gone through the learning activities. After you have passed the exam it is time to move onto a new challenge. The content required for this badge has quite a bit of overlap with that of the Deploying Cookbooks Badge and the Local Cookbook Development Badge. If you don't already have them, those badges would be good targets to work towards now. If you're looking for a completely different challenge and you enjoy the testing side of things then I would encourage you to start learning more about InSpec and working towards the Auditing with InSpec Badge. I hope that you enjoyed this course and felt that it prepared you well for the exam. If there were any parts of the course that you particularly liked or disliked please go back to those pieces of content (videos, live environments, quiz questions, etc.) and leave a rating so that I can continue to improve this course. Thank you for helping us make better content and trusting us with your valuable time.