Use case scenario 2 – multiple hosts

The Jenkins UI can also be tailored to help in managing installations that require a large numbers of Slave hosts. This may be required to improve the performance of builds or test runs by distributing the load to other systems, or wherever Jenkins is used to perform functions spanning a multiple-host Operating System—something that Jenkins can do very easily through the built-in JNLP functionality.

Often, testing requirements dictate that a wide variety of different nodes running varying combinations of OSes and software are essential—this is common when you have an application that needs testing on different versions of Internet Explorer; each version requires a different Windows host, as each host can only support one version of the browser at a time.

Managing multiple and varying Slave Nodes can be problematic; however, the Jenkins UI provides several features that can help to simplify and automate this aspect.

One of the simplest approaches to manage instances with many Slave nodes is to use a Slave labeling scheme and a naming convention that describes the abilities or functions that the individual node perform.

To do this, you first need to label your Slave nodes—this can be done as and when they are created, or you can go back to existing Slave Nodes and label them as desired—note the multiple labels specified in the following Slave definition page:

As you can see, this simple Slave has been given multiple labels of tomcat, java6, and integration.

We can now create or amend a job and select the Restrict where this project can be run option, as shown in the following screenshot:

If we enter a label that matches one or more existing labels (integration in this instance), then this job will be run on a node matching this label. In cases where there are multiple matches, the job will be run on just one of the nodes.

This simple yet extremely powerful UI feature enables you to distribute the load across multiple nodes. The nodes may perform different functions, or they could be set up with different abilities—the labels can be whatever that helps you to decide what is best for your situation.

You could decide to distinguish between the physical characteristics of your nodes, such as those with a large free disk space, more memory or fast processors, or ones that have local databases or code deployments at the desired level, or with application servers or other supporting tools on them. This way you can not only distribute the load, but also maximize performance and reduce turnaround time by putting the right jobs on the hosts that are best suited for them, and by pooling your resources to fine-tune the response time of the various build tasks—getting the urgent tasks turned around as quickly as possible and leaving the less urgent jobs queued on a pool of dedicated servers.

This approach can be taken further using the Swarm plugin: https://wiki.jenkins-ci.org/display/JENKINS/Swarm+Plugin.

This plugin provides an added functionality that enables you to automatically provision and connect new Slave Nodes via a UDP broadcast that discovers and connects new nodes to the existing Master node, thereby creating an ad hoc cluster that you can tailor to meet demands.

You can use this to set things up so that when the build queue reaches a predefined threshold, new nodes will be dynamically provisioned and added to the pool of available nodes. You need to install the feature on the Master node and use the command-line client on the new Slave nodes.

Swarm nodes can also be assigned multiple labels at creation time through the -labels argument in the CLI. These values can additionally be set by the script that creates them; for example, the script could check for the existence of a local Oracle installation or a high percentage of free disk space and then use these results to decide which labels to apply to itself—database, performance, java6, tomcat, and so on accordingly.

Another very powerful tool for dynamic host provisioning and management is Docker, and, not surprisingly, there is a Jenkins plugin available for this too:

https://wiki.jenkins-ci.org/display/JENKINS/Docker+Plugin

Docker allows you to quickly and easily create and manage Docker Images that run in Docker Containers. These are quite similar in practice to virtual machines, but are smaller and of lighter weight, and therefore far quicker and easier to provision than traditional VMs.

Docker Images can also be version controlled via a Docker Registry, which works like a Git or Subversion repository for virtual machines; you can pull an existing Image from the Docker Index and update it to suit your requirements (as you would for a virtual machine—performing tasks such as deploying Tomcat, installing and configuring Apache, uploading some scripts, adding a version of Java, or installing Jenkins). Once you have customized your image, you can optionally push/publish it back to the index in exactly the same state that you created but under a new name, thus creating a template Slave that you can provision both rapidly and reliably to any platform that runs Docker. You can even run Docker on virtual machines—the possibilities that this approach provides are very interesting, and we will look at this in a little more detail in Chapter 9 Putting Things Together.