index.html

<!DOCTYPE html>
<html>
<head>
    <meta charset="utf-8">
    <title>Code Driven Infrastructure and Deployment</title>
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <meta name="author" content="Fabian Vogler">
    <link rel="stylesheet" href="node_modules/normalize.css/normalize.css">
    <link rel="stylesheet" href="css/doc.css">
    <script src="node_modules/jquery/dist/jquery.min.js"></script>
    <script src="js/doc.js"></script>
</head>
<body>

<div class="front">

<p>
ZHAW Zurich University of Applied Sciences<br/>
Bachelor's degree programme Computer Science<br/>
Bachelor Thesis
</p>
<h1>Code Driven Infrastructure and&nbsp;Deployment</h1>
<p class="cover">
<img src="images/cover.svg" width="400" height="341" alt="" />
</p>
<p>
Author: Fabian Vogler, <a href="mailto:fabian.vogler@gmail.com">fabian.vogler@gmail.com</a><br/>
Student number: 10-966-596<br/>
Advisors: Beat Seeliger, Silvan Spross<br/>
<span class="version">2015-07-31, Version 1.1<span id="version-build"></span></span>
</p>

</div>

<div class="chapter">

<p>
This thesis was done as part of a bachelor's degree study at ZHAW Zurich University of Applied Sciences in Zurich. The source code of this document is available online at <a href="https://github.com/fabian/code-driven-infrastructure-and-deployment">https://github.com/fabian/code-driven-infrastructure-and-deployment</a>.
</p>

<p>
All sources are numbered [<em>n</em>] and listed in the bibliography in the appendix. Basic knowledge in computer science is required for reading and understanding the thesis. The most important acronyms and concepts are explained in the glossary that can be found in the appendix as well.
</p>

<p>
This document has been written in HTML and was converted into a PDF document with <a href="#ref-prince">Prince</a>. The font used is <em>Helvetica Neue</em>, created by D.&nbsp;Stempel&nbsp;AG and based on <em>Helvetica</em> by Max Miedinger.
</p>

<p>
The icons used on the cover and in this document are from <a href="#ref-small-n-flat">small-n-flat</a>, they are released to the public domain.
</p>

</div>


<div class="chapter">

<h2 class="break nonr">Abstract</h2>

<p>
WorldSkills International manages its web infrastructure manually with a small team of developers. There is no designated system administrator, the responsibility of managing the web infrastructure is shared among the developers. The author of this thesis is employed as a developer by WorldSkills International. Target of the thesis is to develop a concept for a testable and reproducible infrastructure where all changes are done within a revision control system. This increases visibility of changes within the developers team and makes the change tracable.
</p>

<p>
Many software solutions exist for provisioning of servers and deployment of applications. They can be grouped into software containers, configuration repository, and remote command execution. With software containers the software needed to run an application is encapsulated and run in operating system-level virtualization. A configuration repository is a centralized repository with configuration files and software is used to configure servers based on the configuration files. With remote command execution the installation commands and configuration files are transmitted in a coordinated manner to a remote server.
</p>

<p>
The current infrastructure requirements were analyzed and documented. The different types of software for provisioning of servers and deployment of applications were considered for solving the problem. Each type was evaluated with a popular representative according to the requirements. The evaluation showed that software containers need additional software for the orchestration of the provisioning and deployment. Due to the additional complexity and the missing requirements for software containers their usage was delayed to a potential separate project after this thesis. The automation software Ansible was choosen as best fitting for the requirements.
</p>

<p>
An architecture concept for an automated infrastructure was developed and successfully verified in a proof-of-concept. All software and configuration files required are defined in structured text files that can be read and transmitted to the server with Ansible. Local development and testing can be done in a virtual machine on the developers’ computer. The whole infrastructure can easily be cloned by running the provisioning scripts against new servers. A continious integration server executes the provisioning script after each change and verifies the infrastructure with system tests. The automated creation of the proof-of-concept infrastructure takes about 20 minutes. The automated process of setting up a new infrastructure environment for testing is simply initiated by creating a new branch adhering to a naming scheme. The implementation of the architecture concept and a migration to the new automated infrastructure is planned for fall 2015.
</p>

</div>


<div class="chapter">

<h2 class="break nonr">Contents</h2>
<ol id="toc"></ol>

</div>


<div class="chapter">

<h2 class="break">Introduction</h2>

<h3>Current Situation</h3>

<p>
WorldSkills International is a non-profit membership association which organizes a world championship in skilled professions every two years. The author of this thesis is employed by WorldSkills International. To manage members and to organize the preparation and execution of the competition the organization runs multiple web applications. The mix of PHP and Java applications consists of legacy systems and a newly developed software system with a service-oriented architecture.
</p>

<figure class="full">
<img src="images/application_screenshots.svg" />
<figcaption>Screenshots applications</figcaption>
</figure>

<p>
All web software is running on rented virtual servers with a Linux operating system. They are managed manually using a web control panel (<a href="#ref-plesk">Parallels Plesk</a>). Changes to the infrastructure are done by the four internal software developers manually.
</p>

<p>
Fundamental changes like the migration to a new server or the switch to a new runtime engine require a lot of knowledge about the existing system and manual testing of the new installation.
</p>

<h3>Objectives</h3>

<p>
The main goal of this thesis is to develop a concept for a versioned, testable and reproducible infrastructure. Changes to the system should be visible for the IT team and traceable if needed. As a result knowledge is shared in written form.
</p>

<p class="break">
To achieve this goal, manual steps to build or change the infrastructure should be replaced by code stored in a revision control system. Three different types of software for provisioning of servers and deployment exist at the moment:
</p>

<ul>
    <li>Software containers: Software needed to run an application is encapsulated and run in operating system-level virtualization. Examples are <a href="#ref-docker">Docker</a>, <a href="#ref-rkt">rkt</a>.</li>
    <li>Configuration repository: A centralized repository with configuration files and software is used to configure servers. Examples are <a href="#ref-chef">Chef</a>, <a href="#ref-puppet">Puppet</a>, <a href="#ref-cfengine">CFEngine</a>.</li>
    <li>Remote command execution: Installation commands and configuration files are transmitted in a coordinated manner to a remote server. Examples are <a href="#ref-ansible">Ansible</a>, <a href="#ref-saltstack">SaltStack</a>, <a href="#ref-rex">Rex</a>.</li>
</ul>

<p>
These types of software should be evaluated and an architecture documentation as well as a test concept should be written. The proposed architecture should then be tested in a proof-of-concept.
</p>

<h3>Tasks</h3>

<p>
The following tasks will be completed by the student as part of the bachelor thesis:
</p>

<ol>
    <li>Analyze infrastructure requirements</li>
    <li>Test requirements with popular representatives for each type of provisioning software</li>
    <li>Evaluate provisioning software</li>
    <li>Write architecture documentation</li>
    <li>Develop test concept</li>
    <li>Implement proof-of-concept with automated provisioning and deployment</li>
</ol>

<h3 class="break">Project Management</h3>

<p>
The following gantt diagram shows an overview of the actual timeline during the project as well as the dates of the most important milestones.
</p>

<figure class="full">
<table>
    <colgroup>
        <col style="width: 180px;" />
        <col style="width: 21px;" />
        <col style="width: 21px;" />
        <col style="width: 21px;" />
        <col style="width: 21px;" />
        <col style="width: 21px;" />
        <col style="width: 21px;" />
        <col style="width: 21px;" />
        <col style="width: 21px;" />
        <col style="width: 21px;" />
        <col style="width: 21px;" />
        <col style="width: 21px;" />
        <col style="width: 21px;" />
        <col style="width: 21px;" />
        <col style="width: 21px;" />
        <col style="width: 21px;" />
        <col style="width: 21px;" />
        <col style="width: 21px;" />
        <col style="width: 21px;" />
        <col style="width: 21px;" />
        <col style="width: 21px;" />
        <col style="width: 21px;" />
        <col style="width: 21px;" />
        <col style="width: 21px;" />
        <col style="width: 21px;" />
    </colgroup>
    <thead>
        <tr>
            <th>&#160;</th>
            <th colspan="4">February</th>
            <th colspan="4">March</th>
            <th colspan="4">April</th>
            <th colspan="4">May</th>
            <th colspan="4">June</th>
            <th colspan="4">July</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td>Kick-Off</td>
            <td colspan="4" style="text-align: right;">25.02.15&nbsp;◆</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
        </tr>
        <tr>
            <td>Requirements analysis</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td colspan="4" style="background-color: #e8f0f8;">&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
        </tr>
        <tr>
            <td>Software evaluation</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td colspan="3" style="background-color: #e8f0f8;">&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
        </tr>
        <tr>
            <td>Design Review</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td colspan="4" style="border-left: 0;">◆&nbsp;29.04.15</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
        </tr>
        <tr>
            <td>Architecture concept</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td colspan="6" style="background-color: #e8f0f8;">&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
        </tr>
        <tr>
            <td>Test concept</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td colspan="6" style="background-color: #e8f0f8;">&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
        </tr>
        <tr>
            <td>Proof-of-concept</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td colspan="3" style="background-color: #e8f0f8;">&#160;</td>
            <td>&#160;</td>
        </tr>
        <tr>
            <td>Final date</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td colspan="4" style="text-align: right;">31.07.15&nbsp;◆</td>
        </tr>
    </tbody>
</table>
<figcaption>Project schedule</figcaption>
</figure>

<p>
According to the <a href="#ref-rules-bachelor-thesis">rules</a> at least 360 hours have to be invested into the bachelor thesis. The planning was done according to those hours.
</p>

<p>
Both the requirements analysis and the software evaluation took longer than expected as many aspects had to be studied in detail. In return the architecture  concept could be done faster. The proof-of-concept also took less time then expected as only a small number of problems occured during development. Writing the documentation was slightly underestimated.
</p>

<figure>
<table class="full">
    <colgroup>
        <col class="col-fifty" />
        <col />
        <col />
    </colgroup>
    <thead>
        <tr>
            <th>Description</th>
            <th>Planned</th>
            <th>Actual</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td>Requirements analysis</td>
            <td>64 h</td>
            <td>~72 h</td>
        </tr>
        <tr>
            <td>Software evaluation</td>
            <td>48 h</td>
            <td>~64 h</td>
        </tr>
        <tr>
            <td>Architecture concept</td>
            <td>88 h</td>
            <td>~80 h</td>
        </tr>
        <tr>
            <td>Test concept</td>
            <td>40 h</td>
            <td>~40 h</td>
        </tr>
        <tr>
            <td>Proof-of-concept</td>
            <td>80 h</td>
            <td>~64 h</td>
        </tr>
        <tr>
            <td>Write documentation</td>
            <td>40 h</td>
            <td>~56 h</td>
        </tr>
        <tr class="total">
            <td>Total</td>
            <td>360 h</td>
            <td>376 h</td>
        </tr>
    </tbody>
</table>
<figcaption>Comparison planned and actual hours</figcaption>
</figure>

</div>


<div class="chapter">

<h2 class="break">Requirements</h2>

<h3>Overview</h3>

<p>
The following requirements were established by analysing the existing infrastructure and taking known problems into account.
</p>

<h3>Stakeholders</h3>

<p>
The table below shows the stakeholders which have been found for the current infrastructure. They have a direct or indirect influence on the requirements. These stakeholders are also used in the system context diagram.
</p>

<figure>
<table>
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Stakeholder</th>
<th>Description</th>
</tr>
<tr>
<th>
Developer
</th>
<td>
<p>
Works at WorldSkills International and is responsible for the development and maintenance of the infrastructure. There are four developers working full-time for WorldSkills International. There is no designated system administrator.
</p>
<p>
Developers have different backgrounds and therefore different knowledge about specific components of the instrastructure. They all share the responsibility for keeping the infrastructure running.
</p>
<p>
All developers are using Mac OS X for development. They work from three different time zones.
</p>
</td>
</tr>
<tr>
<th>
User
</th>
<td>
<p>
Interacts with applications running on the WorldSkills infrastructure. This includes the Secretariat, Members, competition personnel and website visitors. Most of them are registered users. Their expectations for fast and continuously running services influence the requirements.
</p>
</td>
</tr>
<tr>
<th>
Hosting provider
</th>
<td>
<p>
Provides virtual Linux servers for the infrastructure. As they are in a highly competitive and fast moving market they can become obsolete and need to be replaced with another hosting provider with a better offering.
</p>
</td>
</tr>
<tr>
<th>
<a href="#ref-github">GitHub</a>
</th>
<td>
<p>
Hosts Git code repositories for WorldSkills International. Provides a web interface for managing permissions of the repositories. They control how code can be accessed.
</p>
</td>
</tr>
<tr>
<th>
<a href="#ref-codeship">Codeship</a>
</th>
<td>
<p>
Provides a hosted continuous integration software for WorldSkills International. The software is based on Linux with support for PHP, Java and JavaScript applications. Their functionality defines how applications can be built and deployed.
</p>
</td>
</tr>
</tbody>
</table>
<figcaption>Stakeholders</figcaption>
</figure>


<h3>System Context</h3>

<figure>
<img src="images/system-context.svg" />
<figcaption>System context</figcaption>
</figure>

<p>
Both Developer and User need to interact with the infrastructure or the applications running on it. They have a direct influence on the requirements and lie within the system context.
</p>

<p>
The source code of most applications is stored on GitHub. Codeship is used for running automated tests and executing the deployment of new versions. The hosting provider supplies the servers for running the infrastructure. All three vendors influence the requirements indirectly with the constraints of their services. They are outside of the system context as they cannot be influenced.
</p>


<h3 class="break">Applications</h3>

<p>
The current infrastructure is composed of multiple applications deployed on three servers. There is no requirement to keep them on separate servers. The following diagram gives an overview of all applications. A short description of each application can be found in the appended table.
</p>

<figure>
<img src="images/overview.svg" />
<figcaption>System overview</figcaption>
</figure>

<p class="break">
The following table lists all applications and their special requirements.
</p>

<figure class="full">
<table>
<colgroup>
    <col class="col-twenty" />
    <col class="col-fifty" />
    <col />
    <col />
    <col />
    <col />
</colgroup>
<tbody>
<tr>
<th rowspan="2">Application</th>
<th rowspan="2">Description</th>
<th colspan="4">Requirements</th>
</tr>
<tr>
<th>RabbitMQ</th>
<th>MySQL</th>
<th>JavaMail</th>
<th>Uploads</th>
</tr>
<tr>
<th>
Web services
</th>
<td>
Java applications for managing organization information
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="Yes">
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="Yes">
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="Yes">
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="Yes">
</td>
</tr>
<tr>
<th>
Management
</th>
<td>
JavaScript applications for accessing the web services
</td>
<td>
</td>
<td>
</td>
<td>
</td>
<td>
</td>
</tr>
<tr>
<th>
Auth
</th>
<td>
PHP application for login
</td>
<td>
</td>
<td>
</td>
<td>
</td>
<td>
</td>
</tr>
<tr>
<th>
worldskills.org
</th>
<td>
Organization website
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="Yes">
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="Yes">
</td>
<td>
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="Yes">
</td>
</tr>
<tr>
<th>
WSC2015 website
</th>
<td>
WorldSkills São Paulo 2015 event website
</td>
<td>
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="Yes">
</td>
<td>
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="Yes">
</td>
</tr>
<tr>
<th>
WSC2017 website
</th>
<td>
WorldSkills Abu Dhabi 2017 event website
</td>
<td>
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="Yes">
</td>
<td>
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="Yes">
</td>
</tr>
<tr>
<th>
Members map
</th>
<td>
World map with Facebook pages from other countries
</td>
<td>
</td>
<td>
</td>
<td>
</td>
<td>
</td>
</tr>
<tr>
<th>
IL
</th>
<td>
PHP application for managing infrastructure lists
</td>
<td>
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="Yes">
</td>
<td>
</td>
<td>
</td>
</tr>
<tr>
<th>
Aggregator
</th>
<td>
PHP application which serves the mobile app content
</td>
<td>
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="Yes">
</td>
<td>
</td>
<td>
</td>
</tr>
<tr>
<th>
Who-is-who
</th>
<td>
PHP application for managing organization personnel
</td>
<td>
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="Yes">
</td>
<td>
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="Yes">
</td>
</tr>
<tr>
<th>
Registrations
</th>
<td>
PHP application for registering people and a web service
</td>
<td>
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="Yes">
</td>
<td>
</td>
<td>
</td>
</tr>
<tr>
<th>
Forums
</th>
<td>
Dicussion forums
</td>
<td>
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="Yes">
</td>
<td>
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="Yes">
</td>
</tr>
<tr>
<th>
Portal
</th>
<td>
Website with information about WorldSkills Competition participants
</td>
<td>
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="Yes">
</td>
<td>
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="Yes">
</td>
</tr>
<tr>
<th>
CIS demo
</th>
<td>
Competition Information System demo
</td>
<td>
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="Yes">
</td>
<td>
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="Yes">
</td>
</tr>
<tr>
<th>
Rooms
</th>
<td>
Java application for reserving meeting rooms
</td>
<td>
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="Yes">
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="Yes">
</td>
<td>
</td>
</tr>
<tr>
<th>
Archive
</th>
<td>
Static copies of old event websites
</td>
<td>
</td>
<td>
</td>
<td>
</td>
<td>
</td>
</tr>
<tr>
<th>
Mailer
</th>
<td>
PHP application for sending emails to groups of people
</td>
<td>
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="Yes">
</td>
<td>
</td>
<td>
</td>
</tr>
<tr>
<th>
SMP
</th>
<td>
Skill Management Plan for planning the skill competitions
</td>
<td>
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="Yes">
</td>
<td>
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="Yes">
</td>
</tr>
<tr>
<th>
CPT
</th>
<td>
Competition Planning Timetable with important deadlines
</td>
<td>
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="Yes">
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="Yes">
</td>
<td>
</td>
</tr>
</tbody>
</table>
<figcaption>Application requirements</figcaption>
</figure>


<h3 class="break">User Stories</h3>

<p>
To describe the functional requirements for the new infrastructure user stories are used in this chapter. They are prioritized in agreement with the developers in three levels: Must, Should, Could.
</p>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Name</th>
<th>R01 PHP applications</th>
</tr>
<tr>
<th>
Description
</th>
<td>
As a developer I want to run multiple PHP applications so users can access them. A PHP application usually needs a MySQL database, the source code is stored on GitHub.
</td>
</tr>
<tr>
<th>
Acceptance Criteria
</th>
<td>
Each PHP application is running and can be accessed with a web browser.
</td>
</tr>
<tr>
<th>Priority</th>
<td>Must</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Name</th>
<th>R02 Java applications</th>
</tr>
<tr>
<th>
Description
</th>
<td>
As a developer I want to run multiple Java applications on a Tomcat server so users and other applications can use the services provided by the applications. A Java application usually needs a MySQL database, the source code is stored on GitHub.
</td>
</tr>
<tr>
<th>
Acceptance Criteria
</th>
<td>
Each Java application is running and can be accessed over HTTP/S.
</td>
</tr>
<tr>
<th>Priority</th>
<td>Must</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Name</th>
<th>R03 Server configuration</th>
</tr>
<tr>
<th>
Description
</th>
<td>
As a developer I might want to change a server configuration file to optimize a setting. For example the MySQL Query Cache size needs to be increased. Another example would be that the servlet container configuration needs a new variable due to a change in the application.
</td>
</tr>
<tr>
<th>
Acceptance Criteria
</th>
<td>
A configuration file gets modified and the change is pushed to the code repository. The new configuration file is automatically deployed to the server and the affected applications loads the new configuration.
</td>
</tr>
<tr>
<th>Priority</th>
<td>Must</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Name</th>
<th>R04 Application deployment</th>
</tr>
<tr>
<th>
Description
</th>
<td>
As a developer I want to deploy a new version of an application so users can benefit from new features or bug fixes. The deployment causes no downtime of the application.
</td>
</tr>
<tr>
<th>
Acceptance Criteria
</th>
<td>
A new version of an application gets pushed to the code repository. Automated tests of the application are executed and if they pass the application gets deployed to the server. The old version keeps responding to requests until the new version is ready.
</td>
</tr>
<tr>
<th>Priority</th>
<td>Should</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Name</th>
<th>R05 Staging environment</th>
</tr>
<tr>
<th>
Description
</th>
<td>
As a developer, I want to test a new version of a web service in a staging environment so I can make sure it works correctly with all other components of the system.
</td>
</tr>
<tr>
<th>
Acceptance Criteria
</th>
<td>
A new version of a web service is pushed in a separate branch, the functionality is available for testing in a staging environment within 30&nbsp;minutes from the push.
</td>
</tr>
<tr>
<th>Priority</th>
<td>Should</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Name</th>
<th>R06 Hosting provider switch</th>
</tr>
<tr>
<th>
Description
</th>
<td>
As a developer I want to switch the server hosting provider so I can profit from a better offer. Another reason could be that the current hosting provider shuts down or its no longer justifiable because of his actions (e.g. security problems).
</td>
</tr>
<tr>
<th>
Acceptance Criteria
</th>
<td>
The infrastructure can be ported to a different provider within 48 hours. All needed software is installed automatically, databases and user files are transferred manually.
</td>
</tr>
<tr>
<th>Priority</th>
<td>Could</td>
</tr>
</tbody>
</table>
</figure>


<h3 class="break">Non-functional Requirements</h3>

<p>
The following non-functional requirements are not conclusive but they are the most critical ones. They are classified according to quality model of <a href="#ref-iso-25010">ISO/IEC 25010:2011</a> and prioritized in agreement with the developers in three levels: Must, Should, Could.
</p>

<p>
The requirements are based on the current infrastructure and events in the past related to it.
</p>


<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Name</th>
<th>R11 Configuration files</th>
</tr>
<tr>
<th>
Classification
</th>
<td>
Changeability
</td>
</tr>
<tr>
<th>
Description
</th>
<td>
All configuration files are stored in a code repository.
</td>
</tr>
<tr>
<th>Priority</th>
<td>Must</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Name</th>
<th>R12 Change history</th>
</tr>
<tr>
<th>
Classification
</th>
<td>
Accountability
</td>
</tr>
<tr>
<th>
Description
</th>
<td>
Every change must be traceable by a developer. Associated with every change is an explanation.
</td>
</tr>
<tr>
<th>Priority</th>
<td>Must</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Name</th>
<th>R13 Open Source</th>
</tr>
<tr>
<th>
Classification
</th>
<td>
Replaceability
</td>
</tr>
<tr>
<th>
Description
</th>
<td>
All software used for infrastructure has to be built on open-source software. This guarantees that components can easily be ported to different providers or maintainers. It also allows other Members to easily copy parts of the infrastructure.
</td>
</tr>
<tr>
<th>Priority</th>
<td>Must</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Name</th>
<th>R14 Test environment</th>
</tr>
<tr>
<th>
Classification
</th>
<td>
Testability
</td>
</tr>
<tr>
<th>
Description
</th>
<td>
To test configuration changes to it, the whole or part of the infrastructure can be started in a test environment. This is different from the staging environment in that the test environment can be local and automated tests are executed against it.
</td>
</tr>
<tr>
<th>Priority</th>
<td>Must</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Name</th>
<th>R15 Encrypted passwords</th>
</tr>
<tr>
<th>
Classification
</th>
<td>
Confidentiality
</td>
</tr>
<tr>
<th>
Description
</th>
<td>
Server passwords should be stored only encrypted on third-party systems. The advantage of storing encrypted passwords in the code repository and sharing a key file instead of sharing the passwords in a file, is that the file doesn't need to be updated for everyone each time a password is added.
</td>
</tr>
<tr>
<th>Priority</th>
<td>Must</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Name</th>
<th>R16 Superuser access</th>
</tr>
<tr>
<th>
Classification
</th>
<td>
Technical accessibility
</td>
</tr>
<tr>
<th>
Description
</th>
<td>
In case of a problem that only occurs in a certain environment, a developer needs unrestricted access to the server to debug the error and try out different solutions.
</td>
</tr>
<tr>
<th>Priority</th>
<td>Must</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Name</th>
<th>R17 Custom software</th>
</tr>
<tr>
<th>
Classification
</th>
<td>
Interoperability
</td>
</tr>
<tr>
<th>
Description
</th>
<td>
New software can be installed without restrictions. New features or analytics tools might require the installation of additional software.
</td>
</tr>
<tr>
<th>Priority</th>
<td>Must</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Name</th>
<th>R18 Learning curve</th>
</tr>
<tr>
<th>
Classification
</th>
<td>
Learnability
</td>
</tr>
<tr>
<th>
Description
</th>
<td>
How to use the software system to install and configure the infrastructure can be learned quickly so all developers can make changes to the infrastructure without spending weeks studying it.
</td>
</tr>
<tr>
<th>Priority</th>
<td>Should</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Name</th>
<th>R19 Horizontal scaling</th>
</tr>
<tr>
<th>
Classification
</th>
<td>
Changeability
</td>
</tr>
<tr>
<th>
Description
</th>
<td>
Horizontal scalable applications can be installed on multiple servers and served through a load balancer.
</td>
</tr>
<tr>
<th>Priority</th>
<td>Could</td>
</tr>
</tbody>
</table>
</figure>


</div>


<div class="chapter">

<h2 class="break">Evaluation</h2>

<h3>Introduction</h3>

<p>
Based on the given requirements the following evaluation defines the guiding principles for the architecture concept. It compares the different approaches for configuring and deploying software and how well they fit to the existing environment.
</p>

<p>
Only specific attributes of the approaches and their software is analyzed, general comparisons have been written <a href="#ref-infoworld-comparison">before</a>.
</p>

<p>
Cloud solutions for managing applications exist, but are usually locked in on the vendor and targeted at high volumes. Our scaling requirements are low as the infrastructure needs to serve mainly the Competition and the Members and they are both limited by other resources (e.g. Member budgets).
</p>


<h3>Software Containers</h3>

<p>
This type of software recently became popular with <a href="#ref-docker">Docker</a>. The idea is to use the advantages of a virtual machine (isolation, portability, efficiency) while sharing resources. Docker has been chosen as it has been perceived as the most active project. A similar project is <a href="#ref-rkt">rkt (Rocket) by CoreOS</a>.
</p>

<figure>
<img src="images/docker.png">
<figcaption>Logo Docker</figcaption>
</figure>

<p>
Docker only works on Linux, so for development on Mac OS X the software Boot2Docker is used on the development laptop. Installation instructions are provided in the <a href="#ref-docker-docs">Docker documentation</a> in the chapter <em>Installation for Mac OS X</em>.
</p>

<p>
After installation the virtual machine with Linux running Docker is launched with the following two commands.
</p>

<figure>
<pre><code>$ boot2docker init
$ boot2docker start
Waiting for VM and Docker daemon to start...
.............ooo
Started.
Writing /Users/fabian/.boot2docker/certs/boot2docker-vm/ca.pem
Writing /Users/fabian/.boot2docker/certs/boot2docker-vm/cert.pem
Writing /Users/fabian/.boot2docker/certs/boot2docker-vm/key.pem

To connect the Docker client to the Docker daemon, please set:
    export DOCKER_HOST=tcp://192.168.59.103:2376
    export DOCKER_CERT_PATH=/Users/fabian/.boot2docker/certs/boot2docker-vm
    export DOCKER_TLS_VERIFY=1
</code></pre>
<figcaption>Boot2Docker initialization</figcaption>
</figure>

<p>
Another useful tool provided by Docker is Docker Compose which allows multiple instances to be started based on a configuration file. However its usage in production is not recommended at the moment as it's missing some features for managing running instances.
</p>

<p>
During the following verification <a href="#ref-docker-book"><em>The Docker Book</em></a> was used as a reference.
</p>

<h4>Requirements verification</h4>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R01 PHP applications</th>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
Docker publishes official images for certain applications and programming languages in the <a href="#ref-docker-hub">Docker Hub Repository</a>. There's also an official image for PHP which can be found on <a href="#ref-docker-php">GitHub</a>. Maintained images for <a href="#ref-php-versions">recent PHP versions</a> are provided with built-in Apache or PHP-FPM for running the application as a service.
</p>

<p>
The easiest way to run a PHP application with Docker is to the use <a href="#ref-apache-http">Apache web server</a> so the application itself and static files (JavaScript, CSS, etc.) can both be served from one process.
</p>

<p>
The MySQL server would be started in a separate instance, so an additional Dockerfile for MySQL is needed and then needs to be linked with the container running the PHP application. To simplify this Docker Compose can be used. It's a tool that can start multiple Docker instances based on a configuration file.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R02 Java applications</th>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
For Tomcat official images are available from the Docker Hub Repository as well. Multiple versions for Java 7 and 8 are available.
</p>

<p>
Configuration and the application code can be copied to the working directory of Tomcat - the application is then started automatically. Multiple applications can be bundled into one container or one container per application can be used. Again as with PHP, a MySQL server would be started in a separate instance.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R03 Server configuration</th>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
To guarantee reproducible containers, configuration files are usually copied into the container at build time. This allows the container to be tested in a staging environment and guarantees the same results in the production environment.
</p>

<p>
Docker provides the tools to build the image and upload it to a central repository, however it doesn't provide any tools to do an actual deployment of the image to a live environment out of the box.
</p>

</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Fail <img src="images/sign-error.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R04 Application deployment</th>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
There are different ways to implement this with Docker. One way would be to build a new image, deploy it separately and then route requests with a load balancer to the new image. However as noted before, Docker doesn't provide tools to do a deployment of the image nor the orchestration of running machines.
</p>

<p>
Another possibility would be to use volumes to deploy the new application into the running container but this would require each application to have built-in support for zero downtime deployment, which is not always the case (Tomcat supports this with parallel deployment but not PHP).
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Fail <img src="images/sign-error.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R05 Staging environment</th>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
Containers can be started quickly in a new environment (e.g. a new virtual server). Database fixtures can be loaded from additional containers which are linked to the database container.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R06 Hosting provider switch</th>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
Thanks to virtualization the only requirement for a new hosting provider would be to run Docker. Transfer of the data is possible by launching additional containers on the old server and mounting the data volumes. The installation of Docker would need to be done manually if Docker is not pre-installed by the hosting provider. But more importantly the configuration of the host machine (e.g. SSH keys, logging, security settings) could not be automated with Docker alone.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Fail <img src="images/sign-error.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R11 Configuration files</th>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
All configuration files can be stored in the code repository and then added to the container during build time.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R12 Change history</th>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
By storing all Dockerfiles and configuration files in a code repository and building the container images on a continuous integration server each change can be traced back.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R13 Open Source</th>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
Both Docker and the registry server for Docker are Open Source on GitHub, they are licensed under the <a href="#ref-apache-license">Apache License</a>.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R14 Test environment</th>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
Multiple containers can be launched locally so tests can be executed.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R15 Encrypted passwords</th>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
Secret data is usually passed as environment variables to the Docker container. Docker doesn't provide functionality for encrypting information.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Fail <img src="images/sign-error.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R16 Superuser access</th>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
Containers can be accessed from the outside with different methods and superuser access to the host system is required by Docker.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R17 Custom software</th>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
There are some limitations of what can be run inside a container on the kernel level, but they are not affecting most use cases.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R18 Learning curve</th>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
Immutable applications are a complex topic and Docker requires specific methods for normal tasks.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Fail <img src="images/sign-error.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R19 Horizontal scaling</th>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
Multiple instances of images can be started in parallel and Docker Swarm, a clustering tool for Docker, can be used to manage them.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<h4 class="break">Summary Software Containers</h4>

<p>
One notable property of Software Containers is their iteration speed. Due to how file changes are saved by the file system, setup commands don't need to be repeated each time but the resulting files can be restored within seconds. This makes experimenting with the container much faster than setting up a complete virtual machine each time.
</p>

<p>
It became clear that Docker itself is a great tool for reducing the overhead of virtual machines, however it is not a complete solution for infrastructure management (yet).
</p>

<figure>
<table class="full">
<colgroup>
    <col class="col-fifty" />
    <col class="col-fifty" />
</colgroup>
<tbody>
<tr>
<th>Advantages</th>
<th>Disadvantages</th>
</tr>
<tr>
<td>
<ul>
<li>Active Docker community</li>
<li>Software can be updated for each container independently</li>
<li>Clear separation of concern</li>
<li>Can be scaled well horizontally</li>
<li>Fast development iterations</li>
</ul>
</td>
<td>
<ul>
<li>Added complexity</li>
<li>Additional virtualization layer makes debugging harder</li>
<li>New technology, still in development</li>
</ul>
</td>
</tr>
</tbody>
</table>
<figcaption>Analysis Software Containers</figcaption>
</figure>

<p>
A comparison of all requirements can be found after the evaluation of the three different software types.
</p>

<h3 class="break">Configuration Repository</h3>

<p>
<a href="#ref-chef">Chef</a> and <a href="#ref-puppet">Puppet</a> are the most popular representatives for this type of software. It has existed for a few years now and has the target to provide a central repository for infrastructure configuration from which server clients regularly pull their changes. This kind of software type usually also provides a way to use it without a central server but that loses the advantage of having a central point to coordinate changes.
</p>

<figure>
<img src="images/chef.png">
<figcaption>Logo Chef</figcaption>
</figure>

<p>
Chef is used here because of the already existing knowledge of Ruby and its popularity. It is usually running on a Linux machine, so for development a virtual machine with Linux is required. To simplify the setup of the virtual machine, <a href="#ref-vagrant">Vagrant</a> can be used. Vagrant is software to run virtual machines based on a configuration file. After installing Vagrant, a virtual machine can be started with a Linux image provided by Chef which already has all the required software pre-installed.
</p>

<figure>
<pre><code>$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
    default: The Berkshelf shelf is at "/Users/fabian/.berkshelf/…"
==> default: Sharing cookbooks with VM
==> default: Importing base box 'chef/ubuntu-14.04'...
[…]
==> default: Running provisioner: chef_solo...
    default: Installing Chef (latest)...
Generating chef JSON and uploading...
==> default: Running chef-solo...
==> default: stdin: is not a tty
==> default: […] INFO: Forking chef instance to converge...
==> default: […] INFO: *** Chef 12.2.1 ***
==> default: […] INFO: Chef-client pid: 2030
==> default: […] INFO: Setting the run_list to ["recipe[example-chef::default]"]
==> default: […] INFO: Run List is [recipe[example-chef::default]]
==> default: […] INFO: Run List expands to [example-chef::default]
==> default: […] INFO: Starting Chef Run for example-chef-berkshelf
==> default: […] INFO: Running start handlers
==> default: […] INFO: Start handlers complete.
==> default: […] INFO: Chef Run complete in 0.017230899 seconds
</code></pre>
<figcaption>Vagrant initialization with Chef</figcaption>
</figure>

<p>
Chef organizes commands in <em>recipes</em> and <em>cookbooks</em>. Variables are stored in files called <em>data bags</em>. The books <a href="#ref-cooking-chef"><em>Cooking Infrastructure by Chef</em></a> and <a href="#ref-taste-test"><em>Taste Test</em></a> were used during the following verification.
</p>

<h4>Requirements verification</h4>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R01 PHP applications</th>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
Cookbooks for PHP are available in the <a href="#ref-supermarket">Supermarket</a>, the official repository for cookbooks. The PHP cookbook is published and maintained by the company behind Chef itself.
</p>

<p>
However the functionality provided by the cookbook is mostly focused on the PHP extension repository <a href="#ref-pear">PEAR</a>, which is not a requirement. A more popular alternative is the <a href="#ref-apache2-cookbook">apache2 cookbook</a>. It provides the possibility to install the Apache web server with mod_php to run PHP together within a web server.
</p>

<p>
For MySQL there's an official cookbook available as well which installs and starts a MySQL server as required.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R02 Java applications</th>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
The Supermarket also has an official cookbook for Tomcat. Java must be installed separately, again there's a <a href="#ref-java-cookbook">cookbook</a>
available in the Supermarket. The same MySQL cookbook as mentioned before can be used.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R03 Server configuration</th>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
With Chef configuration files are usually built from templates which are stored in the code repository. Once uploaded to the Chef server the client receives them, replaces the variables in them and puts them to the targeted location. In addition the service using the file can be restarted.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R04 Application deployment</th>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
For Tomcat parallel deployment can be used here. The actual deployment to the server is done using the Tomcat Manager App.
</p>

<p>
The <code>deploy</code> directive can be used to update a PHP application to the latest revision without causing any downtime. It keeps a copy of the current version running and uses symlinks to make the latest version active once all dependencies have been installed.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R05 Staging environment</th>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
Chef can be used to setup a virtual machine with the needed environment. Once the virtual machine with the Chef client is running it can pull all information needed to setup the environment from the Chef server.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R06 Hosting provider switch</th>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
The Chef client needs to be installed manually on a new virtual machine if the hosting provider doesn't provide machines with Chef pre-installed. Once the Chef client is installed it can install all required software and configure the machine as needed.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R11 Configuration files</th>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
Configuration files can be stored as templates in the code repository. They get compiled and written to their target destination by Chef.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R12 Change history</th>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
All cookbooks and templates can be stored in a code repository, a continuous integration server uploads them to the Chef server for distribution to the clients.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R13 Open Source</th>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
Chef Client, Chef Server and Chef Development Kit (DK) are Open Source on GitHub, they are licensed under the <a href="#ref-apache-license">Apache License</a>.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R14 Test environment</th>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
The cookbooks can also be used to configure the virtual machine on which the tests are running on.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R15 Encrypted passwords</th>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
Data bags can store variables for the Chef server. Confidential information can be stored in encrypted data bags. The secret key can be shared within the developers over other secure channels.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R16 Superuser access</th>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
No restrictions about access to the server are imposed by Chef.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R17 Custom software</th>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
Packages from the Linux distribution are simple to install with Chef, but also custom software can be downloaded and installed.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R18 Learning curve</th>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
Chef has a complex architecture and many dependencies on other libraries. Developers not familiar with Ruby also need to learn the syntax first.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Fail <img src="images/sign-error.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R19 Horizontal scaling</th>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
Multiple Chef clients can connect to the same Chef server and the same cookbooks can be installed on multiple hosts.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<h4 class="break">Summary Configuration Repository</h4>

<p>
The installation of the Chef DK was difficult as it relies on overriding <a href="#ref-rbenv">rbenv</a>, a tool for managing Ruby versions, in the <code>PATH</code> variable. Some modifications to
<code>.bash_profile</code> were needed to get it running properly.
</p>

<p>
The possibility to test the cookbooks locally inside a virtual machine with Vagrant proved handy. However a few times the cookbook failed unexpectedly (e.g. because a wrong package was accidentally installed first) - a complete rebuild solved the problem but took quite some time.
</p>

<p>
A popular cookbook that was initially selected for deploying Java applications turned out to be incompatible with the latest version of the official Tomcat cookbook for no obvious reason. In general the whole system seemed complex:
many components rely on others and compatibility has to be figured out manually.
</p>

<figure>
<table class="full">
<colgroup>
    <col class="col-fifty" />
    <col class="col-fifty" />
</colgroup>
<tbody>
<tr>
<th>Advantages</th>
<th>Disadvantages</th>
</tr>
<tr>
<td>
<ul>
<li>Can be used to manage large number of servers</li>
<li>Utilizes operating system packages</li>
<li>Central server as single source for configuration</li>
</ul>
</td>
<td>
<ul>
<li>Complex dependencies</li>
<li>Maintained server or hosted solution for Chef server needed</li>
<li>Complete rebuild is slow</li>
</ul>
</td>
</tr>
</tbody>
</table>
<figcaption>Analysis Configuration Repository</figcaption>
</figure>


<h3 class="break">Remote Command Execution</h3>

<p>
This type of software is similar to the Configuration Repository described in the previous chapter: Software to be installed and commands to be executed are defined in text files. The main difference is that instead of having a central server where everything is available to get pulled, everything gets pushed to the hosts. In fact Configuration Repository software like Chef also supports this kind of operation mode with Chef Solo.
</p>

<p>
<a href="#ref-ansible">Ansible</a>, <a href="#ref-saltstack">Salt</a>, and <a href="#ref-rex">Rex</a> are implementations specific for this kind of software type. Ansible is used here because it is only focused on this kind of software type and because of its popularity.
</p>

<figure>
<img src="images/ansible.png">
<figcaption>Logo Ansible</figcaption>
</figure>

<p>
Ansible is a Python application and can be installed on a development machine as a Python package according to the <a href="#ref-ansible-docs">Ansible documentation</a>. Ansible uses SSH to communicate to remote servers and execute commands on them. Directives are organized in <em>playbooks</em> and <em>roles</em>.
</p>

<p>
Again Vagrant can be used to simplify the setup of a remote host and test the playbook. As Ansible requires no specific software on the host any Linux distribution can be used with Vagrant. The following snippet shows the initialization process of Vagrant with Ansible:
</p>

<figure>
<pre><code>$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Running provisioner: ansible...
[…] ansible-playbook […] site.yml

PLAY [all] ********************************************************************

[…]

PLAY RECAP ********************************************************************
default                    : ok=1    changed=0
</code></pre>
<figcaption>Vagrant initialization with Ansible</figcaption>
</figure>

<p>
The book <a href="#ref-taste-test"><em>Taste Test</em></a> was used during the following verification.
</p>

<h4>Requirements verification</h4>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R01 PHP applications</th>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
Ansible includes already most tools needed for installing and configuring software using the operating system mechanisms. Examples for how to use them are provided on <a href="#ref-ansible-github">GitHub by Ansible itself</a>.
</p>

<p>
The Apache web server and mod_php can be installed with the operating system packages, their configuration can be created with the Ansible <em>template</em> directive and the <em>service</em> directive makes sure Apache is running. The source code of the PHP application can either be copied to the host or cloned from Git.
</p>

<p>
MySQL can be installed and configured the same way as Apache.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R02 Java applications</th>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
In the examples provided by Ansible they also describe how to install a standalone instance of Tomcat. The installation package gets downloaded from the Tomcat download server, extracted to an appropriate location, the configuration files are created from templates and Tomcat is started as a service. MySQL can be installed with the operating system packages.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R03 Server configuration</th>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
Configuration files can be either simply copied to the server from the local playbook or templates with variables can be used. Ansible checks if there were any changes in the configuration file and notifies a service to restart if needed. The playbook can also be run from an continuous integration server.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R04 Application deployment</th>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
For Tomcat parallel deployment can be used here again. The actual deployment to the server can be done using the Tomcat Manager App or by using Ansible to copy the file to the server.
</p>

<p>
A combination of the <code>git</code> and <code>file</code> directives can be used to checkout the latest version of the PHP application and activate it with a symlink as soon as all dependencies have been installed.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R05 Staging environment</th>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
As long as Ansible can connect via SSH to a machine it can set it up as needed. Ansible can also launch new virtual machines in the cloud which is useful for quickly starting a staging environment.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R06 Hosting provider switch</th>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
As mentioned before the only requirement on the host machine is the ability for Ansible to connect to it. Once this is provided Ansible can setup the new server as required. The only limitation is that Ansible differentiates between package management systems, so switching from <a href="#ref-apt">Apt</a> (used by Debian) to <a href="#ref-yum">Yum</a> (used by SUSE Linux) would require some code changes, but this switch is outside of the scope of this requirement.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R11 Configuration files</th>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
All configuration files can be stored as templates or raw files in the code repository of the playbook. Ansible copies them to the target server when being executed.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R12 Change history</th>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
The playbook and all its files can be stored in the code repository. A continuous integration server can run the playbook if a change has been made.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R13 Open Source</th>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
Ansible is Open Source on GitHub and licenced under <a href="#ref-gpl-license">GNU General Public License (GPL) v3.0</a>.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R14 Test environment</th>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
The playbook can also be executed locally to configure a test environment on the same machine.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R15 Encrypted passwords</th>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
Ansible provides a tool called Vault to encrypt files with variables. The encrypted file can automatically be encrypted when running the playbook. The password can be stored in a separate file and shared among developers.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R16 Superuser access</th>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
Ansible doesn't restrict how the server can be accessed. In fact Ansible requires that the server can be access using SSH which is the preferred way for developers to access the server as well.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R17 Custom software</th>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
Operating system packages as well as custom downloaded software can be installed using Ansible.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R18 Learning curve</th>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
Ansible files are easy to read and the concept can be understood quickly. Only the YAML syntax requires a bit of learning but most examples already give a good idea how it works.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R19 Horizontal scaling</th>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
Ansible can connect to multiple servers and execute the defined commands there. Servers can also be grouped by their role so that only specific commands are executed on certain servers.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<h4 class="break">Summary Remote Command Execution</h4>

<p>
Ansible was easy to install with the Python package manager <a href="#ref-pip">pip</a> and was instantly ready to use. Because all servers are configured with SSH key authentication already, no additional setup is required to make Ansible usable on them.
</p>

<p>
It is basically just a thin layer above shell scripts but provides all the tools needed for provisioning and maintaining a server (commands, services, configuration files). It lacks a few advanced features for highly complex setups, but the fact that it's easy to learn is useful in a small team with shared responsibilities.
</p>

<figure>
<table class="full">
<colgroup>
    <col class="col-fifty" />
    <col class="col-fifty" />
</colgroup>
<tbody>
<tr>
<th>Advantages</th>
<th>Disadvantages</th>
</tr>
<tr>
<td>
<ul>
<li>Few dependencies</li>
<li>Easy to learn</li>
<li>Utilizes operating system packages</li>
<li>No central server needed</li>
</ul>
</td>
<td>
<ul>
<li>Limited reusability of playbooks</li>
<li>No programming in playbooks possible</li>
<li>System package installation not distribution independent</li>
</ul>
</td>
</tr>
</tbody>
</table>
<figcaption>Analysis Remote Command Execution</figcaption>
</figure>

<h3 class="break">Conclusion</h3>

<figure class="full">
<table>
<colgroup>
    <col class="col-fourty" />
    <col class="col-twenty" />
    <col class="col-twenty" />
    <col class="col-twenty" />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>Software Containers</th>
<th>Configuration Repository</th>
<th>Remote Command Execution</th>
</tr>
<tr>
<td>
R01 PHP applications
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="">
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="">
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="">
</td>
</tr>
<tr>
<td>
R02 Java applications
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="">
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="">
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="">
</td>
</tr>
<tr>
<td>
R03 Server configuration
</td>
<td>
<img src="images/sign-error.svg" width="16" height="16" alt="">
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="">
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="">
</td>
</tr>
<tr>
<td>
R04 Application deployment
</td>
<td>
<img src="images/sign-error.svg" width="16" height="16" alt="">
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="">
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="">
</td>
</tr>
<tr>
<td>
R05 Staging environment
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="">
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="">
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="">
</td>
</tr>
<tr>
<td>
R06 Hosting provider switch
</td>
<td>
<img src="images/sign-error.svg" width="16" height="16" alt="">
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="">
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="">
</td>
</tr>
<tr>
<td>
R11 Configuration files
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="">
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="">
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="">
</td>
</tr>
<tr>
<td>
R12 Change history
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="">
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="">
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="">
</td>
</tr>
<tr>
<td>
R13 Open Source
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="">
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="">
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="">
</td>
</tr>
<tr>
<td>
R14 Test environment
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="">
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="">
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="">
</td>
</tr>
<tr>
<td>
R15 Encrypted passwords
</td>
<td>
<img src="images/sign-error.svg" width="16" height="16" alt="">
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="">
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="">
</td>
</tr>
<tr>
<td>
R16 Superuser access
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="">
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="">
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="">
</td>
</tr>
<tr>
<td>
R17 Custom software
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="">
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="">
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="">
</td>
</tr>
<tr>
<td>
R18 Learning curve
</td>
<td>
<img src="images/sign-error.svg" width="16" height="16" alt="">
</td>
<td>
<img src="images/sign-error.svg" width="16" height="16" alt="">
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="">
</td>
</tr>
<tr>
<td>
R19 Horizontal scaling
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="">
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="">
</td>
<td>
<img src="images/sign-check.svg" width="16" height="16" alt="">
</td>
</tr>
</tbody>
</table>
<figcaption>Requirements comparison</figcaption>
</figure>

<p>
Software Containers don't comply with all <em>Must</em> requirements, a Configuration Repository misses one <em>Should</em> requirement (Learning curve), and Remote Command Execution matches all requirements.
</p>

<p>
Even though Software Containers don't comply with all requirements it is still considered a useful concept if it gets combined with a Configuration Repository or Remote Command Execution for orchestration. The next chapter evaluates the possibility of combining Software Containers with another Configuration Repository or Remote Command Execution.
</p>

<h4>Software Containers in Combination</h4>

<p>
The following rough estimation for introducing a software type alone or together with Software Containers under regular workload also shows the result of the higher complexity of a Configuration Repository compared with Remote Command Execution.
</p>

<figure class="affix">
<table class="full">
<colgroup>
    <col />
    <col />
    <col />
    <col />
    <col />
</colgroup>
<tbody>
<tr>
<th></th>
<th>Training</th>
<th>Setup</th>
<th>Migration</th>
<th>Total</th>
</tr>
<tr>
<td colspan="5"></td>
</tr>
<tr>
<td>
Configuration Repository
</td>
<td>
4 weeks
</td>
<td>
7 weeks
</td>
<td>
3 weeks
</td>
<td>
<strong>14 weeks</strong>
</td>
</tr>
<tr>
<td>
Configuration Repository with<br/>
Software Containers
</td>
<td>
6 weeks
</td>
<td>
10 weeks
</td>
<td>
5 weeks
</td>
<td>
<strong>21 weeks</strong>
</td>
</tr>
<tr>
<td colspan="5"></td>
</tr>
<tr>
<td>
Remote Command Execution
</td>
<td>
2 weeks
</td>
<td>
4 weeks
</td>
<td>
2 weeks
</td>
<td>
<strong>8 weeks</strong>
</td>
</tr>
<tr>
<td>
Remote Command Execution with<br/>
Software Containers
</td>
<td>
5 weeks
</td>
<td>
8 weeks
</td>
<td>
5 weeks
</td>
<td>
<strong>18 weeks</strong>
</td>
</tr>
</tbody>
</table>
<figcaption>Estimation introduction time</figcaption>
</figure>

<p>
The introduction of an orchestration software together with Software Containers could be divided into two phases. In the first phase the orchestration could be set up with a Configuration Repository or Remote Command Execution for the existing infrastructure. In a second phase new applications which fit into the model of Software Containers could be adapted to those.
</p>


<figure class="full">
<table class="full">
    <colgroup>
        <col style="width: 180px;" />
        <col style="width: 21px;" />
        <col style="width: 21px;" />
        <col style="width: 21px;" />
        <col style="width: 21px;" />
        <col style="width: 21px;" />
        <col style="width: 21px;" />
        <col style="width: 21px;" />
        <col style="width: 21px;" />
        <col style="width: 21px;" />
        <col style="width: 21px;" />
        <col style="width: 21px;" />
        <col style="width: 21px;" />
        <col style="width: 21px;" />
        <col style="width: 21px;" />
        <col style="width: 21px;" />
        <col style="width: 21px;" />
        <col style="width: 21px;" />
        <col style="width: 21px;" />
        <col style="width: 21px;" />
        <col style="width: 21px;" />
    </colgroup>
    <thead>
        <tr>
            <th rowspan="2">&nbsp;</th>
            <th colspan="8">Phase 1</th>
            <th colspan="12">Phase 2</th>
        </tr>
        <tr>
            <td colspan="4">Month 1</td>
            <td colspan="4">Month 2</td>
            <td colspan="4">Month 3</td>
            <td colspan="4">Month 4</td>
            <td colspan="4">Month 5</td>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td>Training Orchestration</td>
            <td colspan="3" style="background-color: #e8f0f8;">&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
        </tr>
        <tr>
            <td>Setup Orchestration</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td colspan="3" style="background-color: #e8f0f8;">&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
        </tr>
        <tr>
            <td>Migration Orchestration</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td colspan="2" style="background-color: #e8f0f8;">&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
        </tr>
        <tr>
            <td>Training Software Containers</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td colspan="3" style="background-color: #e8f0f8;">&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
        </tr>
        <tr>
            <td>Setup Software Containers</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td colspan="5" style="background-color: #e8f0f8;">&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
        </tr>
        <tr>
            <td>Migration Software Containers</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td>&#160;</td>
            <td colspan="4" style="background-color: #e8f0f8;">&#160;</td>
        </tr>
    </tbody>
</table>
<figcaption>Project schedule</figcaption>
</figure>

<h4>Recommendation</h4>

<p>
Software Containers add additional complexity and their advantages are not urgently needed for a team of four developers. One of them being able to quickly setting up a development environment, which is not as important as no new developers are expected to join the team within the next four years. The initial architecture concept of this thesis will focus only on phase one based on the requirements and the time available.
</p>

<p>
The evaluation has shown that Remote Command Execution is easier to understand and maintain than a Configuration Repository. Understanding the system easily is crucial as all developers need to be able to make quick changes to the infrastructure. Based on this the infrastructure concept will use Remote Command Execution software to describe phase one mentioned above. The software Ansible will be used based on its popularity and the good experience during the evaluation.
</p>

</div>


<div class="chapter">

<h2 class="break">Concept</h2>

<h3>Introduction</h3>

<p>
The following architecture concept is based on the <a href="#ref-arc42">arc42 template</a>, which provides a guideline for documenting software architecture. Goals, constraints and the scope (requirements related information) have already been documented in the previous chapters. For easier understanding some core concepts of Ansible are outlined in this introduction.
</p>

<p>
Ansible uses the term <em>playbook</em> to describe the collective of configuration files, deployment instructions, and environment specific variables. Each playbook can have multiple <em>roles</em> which allow modular grouping of related files. Best practices how to organize playbooks and roles are provided in the <a href="#ref-ansible-docs">Ansible documentation</a>.
</p>

<figure>
<img src="images/ansible-explained.svg" />
<figcaption>Playbook structure</figcaption>
</figure>

<p>
A playbook can also be described as a better structured shell script that tells the server which commands to execute in which order. Before the commands are executed however, it checks if the command is still required or if it is redundant. For configuration files it can check if the file needs to be updated by comparing the desired content to the actual content on the server.
</p>

<p class="break">
Different environment variables can be grouped in <em>inventory</em> files. An inventory can contain a list of server addresses of an environment as well as configuration variables specific to that environment.
</p>

<figure>
<pre><code>[webservers]
foo.example.org
bar.example.org

[webservers:vars]
proxy_host=proxy.example.org
</code></pre>
<figcaption>Example inventory</figcaption>
</figure>

<p>
The <a href="#ref-yaml">YAML</a> syntax is used in playbooks for defining variables and commands to execute. The following is an example of a file for setting up a webserver.
</p>

<figure>
<pre><code>---
- hosts: webservers
  vars:
    http_port: 80
    max_clients: 200
  remote_user: root
  tasks:
  - name: ensure apache is at the latest version
    apt: name=apache2 state=latest
</code></pre>
<figcaption>Example Ansible file</figcaption>
</figure>

<h3>Solution Strategy</h3>

<p>
The whole infrastructure is described in an Ansible playbook which is shared with a code repository on <a href="#ref-github">GitHub</a>. Different services are split up into roles. Inside the roles provisioning and deployment are separate tasks and can be executed independently.
</p>

<p>
Changes to the infrastructure are pushed to the code repository and then Ansible runs on the continuous integration system <a href="#ref-codeship">Codeship</a> to deploy the changes to the servers. Confidential information like passwords and encryption keys are stored in encrypted Ansible Vaults.
</p>

<p>
For application deployment the playbook is downloaded by the continuous integration system of the application and the required deployment tasks are executed with Ansible to update the application on the server.
</p>

<h3 class="break">Building Block View</h3>

<p>
The files and their organization within the playbook used to maintain the WorldSkills infrastructure are explained below. The most important parts of the playbook are the inventories and the roles, they are displayed in the following overview diagram.
</p>

<figure class="full">
<img src="images/playbook-overview.svg" />
<figcaption>WorldSkills playbook overview</figcaption>
</figure>

<p>
Environment specific variables are stored in different inventories. There are two inventories: <em>prod</em> (for production) and <em>staging</em> (for system tests). As environment specific variables often contain sensitive information they are encrypted with Ansible Vault.
</p>

<p>
The role <em>common</em> is used to execute often used tasks like adding public keys of all developers. Instead of hardcoding all servers in the inventory file, they are created or fetched dynamically with the <em>servers</em> role. Each software has its own role for the installation and the setup of the configuration files. Additionally each self-developed application has a role to define the deployment and the setup of configuration files.
</p>

<p>
The following table show the file structure of the playbook and explains the function of most files and folders. The role <em>apache</em> is used as an example, the structure applies for all roles.
</p>

<figure class="full">
<table class="full">
<colgroup>
    <col class="col-fourty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>File</th>
<th>Description</th>
</tr>
<tr>
<td>
<img src="images/file-empty.svg" width="16" height="16" alt="" class="icon">
site.yml
</td>
<td>
Main playbook file, delegate execution of commands to servers
</td>
</tr>
<tr>
<td>
<img src="images/folder.svg" width="16" height="16" alt="" class="icon">
inventories
</td>
<td>

</td>
</tr>
<tr>
<td class="level-1">
<img src="images/folder.svg" width="16" height="16" alt="" class="icon">
prod
</td>
<td>

</td>
</tr>
<tr>
<td class="level-2">
<img src="images/file-empty.svg" width="16" height="16" alt="" class="icon">
hosts
</td>
<td>
Inventory file with list of servers for production
</td>
</tr>
<tr>
<td class="level-2">
<img src="images/folder.svg" width="16" height="16" alt="" class="icon">
group_vars
</td>
<td>

</td>
</tr>
<tr>
<td class="level-3">
<img src="images/file-empty.svg" width="16" height="16" alt="" class="icon">
all
</td>
<td>
Variables for production (e.g. database password)
</td>
</tr>
<tr>
<td class="level-1">
<img src="images/folder.svg" width="16" height="16" alt="" class="icon">
staging
</td>
<td>

</td>
</tr>
<tr>
<td class="level-2">
<img src="images/file-empty.svg" width="16" height="16" alt="" class="icon">
hosts
</td>
<td>
Inventory file with list of servers for staging
</td>
</tr>
<tr>
<td class="level-2">
<img src="images/folder.svg" width="16" height="16" alt="" class="icon">
group_vars
</td>
<td>

</td>
</tr>
<tr>
<td class="level-3">
<img src="images/file-empty.svg" width="16" height="16" alt="" class="icon">
all
</td>
<td>
Variables for staging environment
</td>
</tr>
<tr>
<td>
<img src="images/folder.svg" width="16" height="16" alt="" class="icon">
roles
</td>
<td>

</td>
</tr>
<tr>
<td class="level-1">
<img src="images/folder.svg" width="16" height="16" alt="" class="icon">
apache
</td>
<td>
Role for the installation and configuration of the webserver
</td>
</tr>
<tr>
<td class="level-2">
<img src="images/folder.svg" width="16" height="16" alt="" class="icon">
defaults
</td>
<td>

</td>
</tr>
<tr>
<td class="level-3">
<img src="images/file-empty.svg" width="16" height="16" alt="" class="icon">
main.yml
</td>
<td>
Default webserver configuration variables (e.g. virtual hosts)
</td>
</tr>
<tr>
<td class="level-2">
<img src="images/folder.svg" width="16" height="16" alt="" class="icon">
files
</td>
<td>

</td>
</tr>
<tr>
<td class="level-3">
<img src="images/file-empty.svg" width="16" height="16" alt="" class="icon">
worldskills.org.crt
</td>
<td>
SSL certificate
</td>
</tr>
<tr>
<td class="level-3">
<img src="images/file-empty.svg" width="16" height="16" alt="" class="icon">
worldskills.org.key
</td>
<td>
SSL key
</td>
</tr>
<tr>
<td class="level-2">
<img src="images/folder.svg" width="16" height="16" alt="" class="icon">
handlers
</td>
<td>

</td>
</tr>
<tr>
<td class="level-3">
<img src="images/file-empty.svg" width="16" height="16" alt="" class="icon">
main.yml
</td>
<td>
Commands for restarting the webserver
</td>
</tr>
<tr>
<td class="level-2">
<img src="images/folder.svg" width="16" height="16" alt="" class="icon">
tasks
</td>
<td>

</td>
</tr>
<tr>
<td class="level-3">
<img src="images/file-empty.svg" width="16" height="16" alt="" class="icon">
vhosts.yml
</td>
<td>
Setup of virtual hosts
</td>
</tr>
<tr>
<td class="level-3">
<img src="images/file-empty.svg" width="16" height="16" alt="" class="icon">
main.yml
</td>
<td>
Installation of the webserver and setup of the configuration files
</td>
</tr>
<tr>
<td class="level-2">
<img src="images/folder.svg" width="16" height="16" alt="" class="icon">
templates
</td>
<td>

</td>
</tr>
<tr>
<td class="level-3">
<img src="images/file-empty.svg" width="16" height="16" alt="" class="icon">
vhost.conf.j2
</td>
<td>
Virtual host configuration file template
</td>
</tr>
</tbody>
</table>
<figcaption>File structure playbook</figcaption>
</figure>

<h3>Runtime View</h3>

<p>
This view explains the dynamic aspects of the playbook execution. In particular the creation of the dynamic inventory and how they work together with the roles are highlighted.
</p>

<figure>
<img src="images/sequence.svg" />
<figcaption>Playbook execution</figcaption>
</figure>

<p>
While running the playbook, the first role that gets executed is the <em>servers</em> role. This role doesn't execute any remote commands but runs locally to connect to the web service of the hosting provider. It either creates and boots the servers if they don't exist yet or just fetches their information. The returned IP addresses are added to the dynamic inventory so following commands can be executed on the servers.
</p>

<p>
Subsequently all other roles are executed. This includes setup of software and applications. The commands are sent to the remote servers by Ansible.
</p>

<h3 class="break">Deployment View</h3>

<p>
In the bigger context the playbook is not executed manually on the developer machine but automated on a continuous integration server.
</p>

<figure class="full">
<img src="images/deployment.svg" />
<figcaption>Deployment overview</figcaption>
</figure>

<p>
Source code is stored in Git repositories. The playbook and each application have all their own private code repository. The continuous integration server  has a build pipeline for each code repository. For each pipeline a unique private SSH key is used. The public keys for those private keys can be exported and are deployed to the code repository provider to guarantee access to the source code. They are also copied to the virtual servers to allow Ansible to connect to the servers via SSH.
</p>

<p>
Each application pipeline uses the playbook to provision the infrastructure and then deploys the application to the targeted environment. The servers required for the production and staging environment are supplied by the hosting provider.
</p>

<p>

</p>

<h3>Design Decisions</h3>

<h4>Playbook organization</h4>

<p>
A fundamental decision of the project is how to organize and structure the playbooks. Each application has its own code repository and allows actions to be executed after a push.
</p>

<p>
Two variations are possible: One code repository with a playbook for the infrastructure and deployment, or a playbook in each application code repository. The first approach has been chosen to easily allow sharing of variables from the infrastructure setup with the application deployment.
</p>

<h4>Usage of inventories</h4>

<p>
Ansible is built to also manage legacy infrastructures with existing servers, so the main purpose of inventories is to provide a list of servers grouped by their function. But Ansible also has built-in server provisioners, which allows to take the approach of an idempotent infrastructure one step further and also create the virtual servers with Ansible as part of the playbook.
</p>

<p>
Usually inventory files in Ansible playbooks contain a list of IP addresses or server names which belong to the infrastructure, but with the proposed concept the only server listed in the inventory file is <em>localhost</em>. This is because the servers get added to the inventory list in memory while running the playbook. If the servers don't exist yet, they get created; if they already exist, their IP addresses are added to the inventory.
</p>

<p>
This approach has the advantage that temporary environments (like staging) can easily be created and existing IP addresses of existing servers don't need to be manually maintained inside an inventory file.
</p>

<h4>Ansible Tower</h4>

<p>
The company behind Ansible distributes also commercial software called <a href="#ref-ansible-tower">Ansible Tower</a> for running playbooks with a user interface. The software provides a web interface with permission management, an audit trail and visualizations.
</p>

<figure class="affix">
<img src="images/screenshot-ansible-tower.png" />
<figcaption>Screenshot Ansible Tower</figcaption>
</figure>

<p>
Ansible Tower was not used for this project due to the high price (<a href="#ref-ansible-tower-pricing">starting at $5000 per year</a>) and because it wouldn't bring many additional benefits. As all four developers responsible for the system are used to working with Git there's no need for a graphical user interface. It would also contradict the principle of having all changes driven by code and it's advantage of being able to easily trace a change.
</p>

<h3 class="break">Technical Risks</h3>

<p>
The following paragraphs outlines potential technical risks of the architecture concept. The factors convenience and security are often conflicting and need to be weighted in each case.
</p>

<p>
One disadvantage of automating the server setup is that the executor needs full administration access to the servers. In the proposed concept the continuous integration server has superuser access to all servers. This leads to two attack scenarios: An attacker could compromise the continuous integration server to gain access to the servers. Or the attacker could compromise the code repository and infiltrate the server through malicious commands. Both <a href="#ref-github-security">GitHub</a> and <a href="#ref-codeship-security">Codeship</a> take appropriate actions to prevent these kind of attacks.
</p>

<p>
As the staging environment is always built by syncing databases from production the risk of data leakage increases as staging environments are often regarded as non-critical. By treating the staging environment as confidentially as the production environment (e.g. encrypting staging passwords), incidents can be prevented.
</p>

<h3 class="break">Testing</h3>

<p>
The testing is divided into multiple stages which all aim to find bugs in the Ansible playbook. Testing of the self-developed applications has to be done by each application individually. All automated tests are executed on the continuous integration server.
</p>

<figure class="full">
<img src="images/testing-flow.svg" />
<figcaption>Testing flow</figcaption>
</figure>

<p>
After each code change a syntax check of the Ansible playbook is performed. The playbook is then executed to create a temporary staging environment. To verify the integrity of the staging environment system tests are performed as a last step. Once the staging environment is ready manual tests can be executed depending on the change made. Testing during development can be done locally with Vagrant.
</p>

<h4>Syntax check</h4>

<p>
Ansible provides two flags when running playbooks which are particularly useful for testing. <code>--syntax-check</code> validates the YAML syntax of the playbook and can be used to discover problems early.
</p>

<p>
The flag <code>--check</code> runs the playbook in a special mode where a <em>dry run</em> is performed. In this mode no changes are made on remote systems, but all tasks are checked if they can be executed. Thanks to this it can easily be checked if the playbook would run or fail (e.g. because of some missing variables). Unfortunately this is not possible in all cases as some modules require some dependencies to be installed (e.g. rabbitmq_plugins).
</p>

<p>
Because of the restrictions of the check mode only the syntax check is performed as an automated test.
</p>

<h4>Staging environment</h4>

<p>
New functionality is developed in feature branches in the application repositories. To test new functionality which requires changes in multiple applications they all need to be deployed to the same staging environment. For this a branch naming convention is defined as following: Each time a branch starting with <code>staging-</code> is created, a new staging environment is automatically created.
</p>

<p>
The first time an application uses the branch name, it boots the required servers and installs all applications from their master branches. Only the application using the branch name is installed from the branch.
</p>

<p>
If another application creates a branch with the same name it will get deployed to the same staging environment. It makes sure the servers in the environment are in the desired state and as applications are only updated during the initial provisioning, the first application won't get overridden with the master branch.
</p>

<p>
After testing is completed and the branch has been merged into the master branch the staging environment can be removed with a separate Ansible script.
</p>

<h4>System tests</h4>

<p>
Simple system tests can be done with the <em>uri</em> module of Ansible. The module allows to send HTTP requests and store the response in a variable. The response can then be checked for certain content.
</p>

<figure>
<pre><code>- name: check Auth API
  action: uri url="https://api.worldskills.org/auth/ping" return_content=yes
  register: uri_response

- name: verify Auth API
  fail: msg="Auth API not reachable"
  when: "'pong' not in uri_response.content"</code></pre>
<figcaption>Example system test</figcaption>
</figure>

<p>
Testing the web interface of the applications also verifies that all underlying software like the Apache web server and PHP are running properly. For each self-developed application at least one system should be created to make sure they are running as expected.
</p>

<h3 class="break">Migration</h3>

<p>
To migrate the existing infrastructure to the new automated infrastructure multiple steps need to be taken. Not only do all applications need to be installed and configured on the new infrastructure but existing data needs to be transferred and DNS entries need to be updated.
</p>

<p>
The following steps should be executed in the order outlined. Some have to be done manually, others can be automated with Ansible.
</p>

<ol>
<li>Reduce DNS time to live (TTL), manual</li>
<li>Create new infrastructure, automated</li>
<li>Stop existing infrastructure, manual</li>
<li>Synchronize database to new infrastructure, automated</li>
<li>Synchronize user files to new infrastructure, automated</li>
<li>Switch DNS entries, manual</li>
<li>Increase DNS time to live (TTL), manual</li>
</ol>

<p>
The first steps are reducing the time DNS entries are cached so the switch to the new server gets propagated faster. The new infrastructure is then created with the main Ansible playbook. Once the new infrastructure is ready the old one can be switched off so no changes can be made anymore by users. To synchronize the database and user files from the existing infrastructure a separate Ansible playbook should be written so it can be executed on its own. After everything has been transferred to the new infrastructure the DNS entries can be changed to point to the new infrastructure and the DNS TTL can be increased again.
</p>

</div>


<div class="chapter">

<h2 class="break">Implementation</h2>

<h3>Proof-of-concept</h3>

<p>
For the proof-of-concept the following reduced infrastructure has been selected to test the feasibility of the concept focusing on the requirements. The proof-of-concept infrastructure includes a PHP application (R01), two Java applications (R02), one JavaScript application, multiple MySQL databases and the RabbitMQ message queue.
</p>

<figure>
<img src="images/proof-of-concept.svg" />
<figcaption>Proof-of-concept infrastructure</figcaption>
</figure>

<p>
The proof-of-concept also includes the dynamic generation of staging environments based on branch naming (R04). The source code for all applications and the Ansible playbook is stored in a separate organization on GitHub. The domain name <em>worldskills.ch</em> is used for the proof-of-concept.
</p>

<p>
To prepare for horizontal scaling (R19) databases are installed on a separate server. Initial data fixtures are imported from the existing WorldSkills infrastructure (using an SSH connection).
</p>

<p>
All Must and Should requirements can be tested with the proof-of-concept infrastructure. The following list includes all requirements that are checked with the proof-of-concept infrastructure.
</p>

<ul>
<li>R01 PHP applications</li>
<li>R02 Java applications</li>
<li>R03 Server configuration</li>
<li>R04 Application deployment</li>
<li>R05 Staging environment</li>
<li>R11 Configuration files</li>
<li>R12 Change history</li>
<li>R13 Open Source</li>
<li>R14 Test environment</li>
<li>R15 Encrypted passwords</li>
<li>R16 Superuser access</li>
<li>R17 Custom software</li>
<li>R18 Learning curve</li>
</ul>

<p>
The following Could requirements are not explicitly tested with the proof-of-concept infrastructure because of their complexity.
</p>

<ul>
<li>R06 Hosting provider switch</li>
<li>R19 Horizontal scaling</li>
</ul>

<h3 class="break">Ansible Playbook</h3>

<p>
The playbook code repository is organized with roles. Each role is responsible for the installation and configuration of the applications. The main playbook delegates the roles for execution to the servers.
</p>

<figure class="affix">
<pre><code>- hosts: database

  roles:
    - mysql
    - rabbitmq

- hosts: api

  roles:
    - tomcat
    - worldskills_api_auth
    - worldskills_api_events

- hosts: web

  roles:
    - apache
    - php
    - worldskills_concrete5
    - worldskills_auth
    - worldskills_events
</code></pre>
<figcaption>Main playbook</figcaption>
</figure>

<h4>System Packages</h4>

<p>
The PHP role is an example of software that gets installed with the package manager of the operating system. The <em>template</em> module of Ansible is used to create the required configuration files. The same approach is used for the installation of MySQL, RabbitMQ, Tomcat and the Apache web server.
</p>

<figure class="affix">
<pre><code>- name: install PHP
  apt: name={{ item }} state=present # install software packages
  with_items:
    - php5
    - libapache2-mod-php5
    - php5-mysql
    - php5-curl
    - php5-gd
  notify: restart apache # restart Apache after PHP installation

- name: put configuration file in place
  template: src=worldskills.ini.j2 dest="{{ php_ext_path }}/worldskills.ini"
  notify: restart apache # restart Apache if configuration changes
</code></pre>
<figcaption>Role php tasks</figcaption>
</figure>

<h4>Virtual Hosts</h4>

<p>
Virtual hosts in the Apache web server are also created with the <em>template</em> module and by linking the configuration files to <code>/etc/apache2/sites-available</code>.
</p>

<figure class="affix">
<pre><code>- name: add configuration files
  template: >
    src="vhost.conf.j2"
    dest="/etc/apache2/sites-available/{{ item.servername }}.conf"
  with_items: vhosts
  notify: restart apache

- name: enable vhosts
  file: >
    src="/etc/apache2/sites-available/{{ item.servername }}.conf"
    dest="/etc/apache2/sites-enabled/{{ item.servername }}.conf" state=link
  with_items: vhosts
  notify: restart apache
</code></pre>
<figcaption>Virtual hosts tasks</figcaption>
</figure>

<h4>PHP and JavaScript Applications</h4>

<p>
Self-developed applications are installed by checking out the source code and then installing all dependencies. To keep the existing application running the source code is installed in a new folder and a symlink pointing to the latest checkout is updated at the end of the installation process.
</p>

<figure class="affix">
<pre><code>- name: clone project
  git: >
    repo="git@github.com:worldskills-infrastructure/worldskills-events.git"
    dest="{{ worldskills_events_path }}/repo"
    version="{{ worldskills_events_branch }}"
    update="{{ worldskills_events_update }}"
  register: worldskills_events_clone

- name: set release path
  set_fact: >
    worldskills_events_release="{{ worldskills_events_path }}/releases/
    {{ worldskills_events_clone.after }}"

- name: export a copy of the repo
  command: >
    git checkout-index -a --prefix="{{ worldskills_events_release }}/"
    chdir="{{ worldskills_events_path }}/repo"
    creates="{{ worldskills_events_release }}"

- name: install npm dependencies
  npm: path="{{ worldskills_events_release }}"

- name: install bower dependencies
  command: >
    bower install chdir="{{ worldskills_events_release }}"
    creates="{{ worldskills_events_release }}/app/bower_components"

- name: create config
  template: >
    src=config.js.j2 dest="{{ worldskills_events_release }}
    /app/scripts/config.js"
  notify: build worldskills_events

- name: link release to current
  file: >
    state=link path="{{ worldskills_events_path }}/current"
    src="{{ worldskills_events_release }}"
</code></pre>
<figcaption>Role Events application tasks</figcaption>
</figure>

<h4 class="break">Java Applications</h4>

<p>
Java applications are deployed using the <a href="#ref-tomcat-maven">Tomcat Maven Plugin</a>. This plugin builds the application and uploads it to the Tomcat server. Again the old version of the application is kept running until the new application has been started and is ready. For Tomcat this is achieved with the built-in functionality for parallel deployment as described in the <a href="#ref-tomcat-documentation">Tomcat Documentation</a>.
</p>

<p>
The current build timestamp is used as a substitute for the version number of the application and configured in the Maven file <em>pom.xml</em>.
</p>

<figure class="affix">
<pre><code>&lt;plugin&gt;
    &lt;groupId&gt;org.apache.tomcat.maven&lt;/groupId&gt;
    &lt;artifactId&gt;tomcat7-maven-plugin&lt;/artifactId&gt;
    &lt;version&gt;2.2&lt;/version&gt;
    &lt;configuration&gt;
        &lt;port&gt;9090&lt;/port&gt;
        &lt;contextFile&gt;src/test/resources/context.xml&lt;/contextFile&gt;
        &lt;url&gt;http://api.worldskills.ch/manager/text&lt;/url&gt;
        &lt;path&gt;/events##${maven.build.timestamp}&lt;/path&gt;
        &lt;server&gt;tomcat&lt;/server&gt;
        &lt;update&gt;true&lt;/update&gt;
        &lt;mode&gt;war&lt;/mode&gt;
    &lt;/configuration&gt;
&lt;/plugin&gt;
</code></pre>
<figcaption>Tomcat Maven Plugin configuration</figcaption>
</figure>

<h4 class="break">Ansible Vault</h4>

<p>
As environment specific variables contain sensitive data like database passwords, they are encrypted with Ansible Vault. The password for the Vault is stored in the file <em>vault_password.txt</em> which is not inside the code repository but shared manually between developers. To prevent the password file from accidentaly being added to the code repository it is explicitly excluded in the file <em>.gitignore</em>.
</p>

<p>
The encrypted file in the inventory can be edited with the following command which uses the password file and opens the configured text editor with the content of the file.
</p>

<figure class="affix">
<pre><code>$ ansible-vault \
    edit \
    --vault-password-file=vault_password.txt \
    inventories/prod/group_vars/all
</code></pre>
<figcaption>Ansible Vault edit command</figcaption>
</figure>

<h4 class="break">Data Synchronization</h4>

<p>
As the CMS of the website heavily depends on content in the database to work properly, the database must be synced from the current infrastructure to the proof-of-concept infrastructure. Additionally this synchronization must be done when creating a staging environment.
</p>

<p>
Because Ansible is using SSH the server which the new database is running on can get an export of the database directly from the production database server.
</p>

<figure class="affix">
<pre><code>- name: get worldskills_concrete5 dump from prod
  command: >
    rsync -ze "ssh -o StrictHostKeyChecking=no"
    root@beuk.worldskills.org:/path/to/database/mysqldump.sql.gz
    /usr/local/worldskills-playbook/worldskills_concrete5.sql.gz
    creates=/usr/local/worldskills-playbook/worldskills_concrete5.sql.gz

- name: import worldskills_concrete5 database
  mysql_db: >
    name="{{ worldskills_api_auth_database_name }}"
    state=import
    target=/usr/local/worldskills-playbook/worldskills_concrete5.sql.gz
</code></pre>
<figcaption>Data synchronization tasks</figcaption>
</figure>

<h3 class="break">Vagrant Setup</h3>

<p>
For local development the playbook can be executed in Vagrant. Ansible is configured as provisioner in Vagrant and points to the main playbook. As the Ubuntu image used by Vagrant has the root user disabled by default, the playbook is executed with the user <em>vagrant</em> and with superuser permissions (sudo). Environment specific variables like URLs are overridden with the <code>extra_vars</code> parameter. To simplify things all server groups are mapped to the default server, so only one virtual machine is needed to run the infrastructure locally.
</p>

<figure>
<pre><code>config.vm.provision "ansible", run: "always" do |ansible|
  ansible.playbook = "site.yml" # WorldSkills playbook
  ansible.raw_arguments = ['--diff'] # show file changes
  ansible.sudo = true # force sudo
  ansible.extra_vars = {
    ansible_ssh_user: "vagrant",
    server_prefix: "",
    worldskills_auth_url: "http://vagrant-auth.worldskills.ch/",
    worldskills_events_url: "http://vagrant-events.worldskills.ch/",
  }
  ansible.groups = {
    "servers" => ["default"],
    "web" => ["default"],
    "api" => ["default"],
    "database" => ["default"],
  }
end
</code></pre>
<figcaption>Vagrant provisioner configuration</figcaption>
</figure>

<h3 class="break">GitHub Setup</h3>

<p>
An independent GitHub organization is used to keep the proof-of-concept separated from existing WorldSkills projects. All projects which are part of the proof-of-concept have been replicated as private repositories in this organization. Most notably is the new code repository <em>worldskills-playbook</em> which contains the Ansible playbook.
</p>

<figure class="full">
<img src="images/screenshot-github.png" />
<figcaption>Screenshot GitHub repositories</figcaption>
</figure>

<p>
Every commit is also displayed in <a href="#ref-slack">Slack</a>, the chat application used by WorldSkills. This guarantees that all developers are aware of the change.
</p>

<figure>
<img src="images/screenshot-slack-github.png" />
<figcaption>Screenshot commit in Slack</figcaption>
</figure>

<h3 class="break">Codeship Setup</h3>

<p>
Codeship knows setup, test and deployment commands. Setup and test commands are always executed, deployment commands can be different for each branch. Deployment commands are only executed if the test commands succeeded. If there's a syntax error in the Ansible playbook, the deployment commands are not executed.
</p>

<p>
The setup commands on Codeship install Ansible and other required software. The Vault password is stored as an environment variable on Codeship. During setup the variable is written to a Vault password file. This approach prevents it from being displayed in the execution logs.
</p>

<figure>
<pre><code>pip install ansible dopy httplib2
echo $VAULT_PASSWORD > vault_password.txt
</code></pre>
<figcaption>Codeship setup commands</figcaption>
</figure>

<p>
The test command then runs the syntax check of the main Ansible playbook file locally. The comma at the end of <code>localhost,</code> tells Ansible to use the passed string as inventory instead of trying to load an inventory file.
</p>

<figure>
<pre><code>ansible-playbook -i localhost, site.yml --syntax-check
</code></pre>
<figcaption>Codeship test commands</figcaption>
</figure>

<p>
With the deployment commands, the master branch gets deployed to the production environment. The production inventory is loaded from the file <code>inventories/prod</code> and the Vault password file created with the setup commands is used for encrypting the production variables.
</p>

<figure>
<pre><code>ansible-playbook \
    -i inventories/prod \
    --vault-password-file=vault_password.txt \
    site.yml
</code></pre>
<figcaption>Codeship deployment commands</figcaption>
</figure>

<p>
All branches starting with <code>staging-</code> are deployed to an environment with the branch name. Codeship provides the current branch name as environment variable <code>$CI_BRANCH</code>. The variable <code>digital_ocean_droplet_prefix</code> gets overridden from the command line with the option <code>-e</code> (additional variables).
</p>

<figure class="affix">
<pre><code>ansible-playbook \
    -i inventories/staging \
    --vault-password-file=vault_password.txt \
    -e "digital_ocean_droplet_prefix=$CI_BRANCH" \
    site.yml
</code></pre>
<figcaption>Codeship deployment commands for staging environment</figcaption>
</figure>

<p>
All commands are logged and reported in the Codeship web interface.
</p>

<figure class="full affix">
<img src="images/screenshot-codeship.png" />
<figcaption>Screenshot Codeship execution</figcaption>
</figure>

<p>
With a webhook the status of every build is also displayed in <a href="#ref-slack">Slack</a>.
</p>

<figure>
<img src="images/screenshot-slack.png" />
<figcaption>Screenshot notification Slack</figcaption>
</figure>

<p>
The danger of a race condition with multiple playbooks running at the same time is avoided by the Codeship plan which allows only one concurrent build.
</p>

<h3 class="break">DigitalOcean Setup</h3>

<p>
The servers for the infrastructure are created by the <em>servers</em> role with Ansible on <a href="#ref-digital-ocean">DigitalOcean</a>. The servers are identified by their name, their IP address is stored for usage in other roles (e.g. <code>digital_ocean_ip_web</code>).
</p>

<figure>
<pre><code>- name: create servers
  digital_ocean:
    command: droplet
    state: present
    name: "{{ item.key }}" # server name
    size_id: 63 # 1 GB memory
    image_id: 11836690 # Ubuntu 14.04
    unique_name: yes # identify server by name
    client_id: "{{ digital_ocean_client_id }}"
    api_key: "{{ digital_ocean_api_key }}"
  register: droplets_response # store DigitalOcean response
  with_dict: # loop over list of servers
    web: {group: web}
    api: {group: api}
    database: {group: database}

- name: store server ips # store IP addresses in variables
  set_fact: "digital_ocean_ip_{{ item.item.key }}={{ item.droplet.ip_address }}"
  with_items: droplets_response.results # loop over all servers
</code></pre>
<figcaption>Role servers</figcaption>
</figure>

<p>
After running the servers role the servers are visible in the DigitalOcean web interface.
</p>

<figure class="full">
<img src="images/screenshot-digitalocean-droplets.png" />
<figcaption>Screenshot DigitalOcean servers</figcaption>
</figure>


<h3 class="break">Challenges</h3>

<p>
During the implementation the following problems occurred and had to be fixed. Outlined are descriptions of the problems, the steps taken, and the final solutions.
</p>

<h4>Hierarchy of group_vars</h4>

<p>
Originally it was planned to use <em>group_vars</em> on the playbook level to define default values for various variables and override the values with environment specific values. However in Ansible <em>group_vars</em> on the playbook level precede inventory variables. This hierarchy was reversed in Ansible 1.7 due to a bug and is still <a href="#ref-ansible-github-9877">controversial</a>.
</p>

<p>
The problem has been solved by moving the default values to the individual roles. This has the disadvantage that default values can only be shared between roles that run on the same server. Hopefully Ansible will introduce default values on the playbook level in the future.
</p>

<h4>Problem SSH connection closed</h4>

<p>
A problem that occurred during testing with DigitalOcean was that sometimes the servers couldn't be created properly. When trying to connect to the just created server via SSH the connection always failed.
</p>

<figure>
<pre><code>ssh root@46.101.135.160
Connection closed by 46.101.135.160
</code></pre>
<figcaption>SSH error message closed connection</figcaption>
</figure>

<p>
After resetting the root password of the server via the DigitalOcean web interface and connecting via their remote control interface it turned out the server was missing the SSH keys required to successfully connect to it. Contacting the support of DigitalOcean didn't bring any new information, they assured that they are not aware of any software problem. As this problem only occurs sometimes it is most likely related to a race condition when deleting a server and creating a new one with the same name. Waiting at least 5 minutes before trying to create the same servers again solved the problem.
</p>

<h4>Problem SSH connection timeout</h4>

<p>
Another similar SSH problem was that sometimes the SSH connection was reset while the Ansible playbook was running. A first part of the playbook was successfully executed on the database servers but the next part which was to be executed on the API servers failed with the following message.
</p>

<figure class="affix">
<pre><code>ssh: connect to host 46.101.255.100 port 22: Connection timed out
Couldn't read packet: Connection reset by peer
</code></pre>
<figcaption>SSH error message timed out connection</figcaption>
</figure>

<p>
Checking the SSH timeout variables (ClientAliveInterval, ClientAliveCountMax, ServerAliveInterval) showed that the both the SSH server and the client are configured correctly. Manual testing of the SSH connection from Codeship to DigitalOcean by keeping the connection idle for some time also worked as expected. The problem could not be reproduced and happened only twice. Most likely a temporary network problem between Codeship and DigitalOcean.
</p>

<h4>Problem MySQL socket</h4>

<p>
After running the playbook repeatedly, PHP was suddenly unable to connect to the MySQL database. Investigating the problem it turned out that the MySQL socket had disappeared. After debugging the issue the problem could be tracked down to a problem in Ubuntu when the command <code>/etc/init.d/mysql restart</code> was executed. Due to a recent change in Ubuntu to upstart scripts this is actually a legacy command and <code>service mysql restart</code> should be executed instead.
</p>

<p>
The main issue is that Ansible isn't using the correct restart command, so instead of relying on the Ansible <code>service</code> module, the correct command is now executed with the <code>shell</code> module directly. A fix for this bug will be available in <a href="#ref-ansible-github-999">Ansible 1.9.2</a>. After the release the <code>service</code> module can be used again.
</p>

<h4>Missed handlers when task failed</h4>

<p>
If a task in an Ansible playbook fails, the execution of the whole playbook stops. Queued handlers are lost if this happens and in the next run the handlers might not get notified again. Because of this a required restart easily gets forgotten and can cause unexpected server states (e.g. configuration file has been updated but a restart is required for loading the updated configuration). Luckily Ansible can be forced to always run handlers, even if a task fails with the setting <code>force_handlers:&nbsp;True</code>. This solved the problem of missed application restarts.
</p>

<h3 class="break">Test Results</h3>

<p>
The whole proof-of-concept infrastructure can be created in 18 minutes. The output of the execution can be found below.
</p>

<figure>
<pre><code class="long">PLAY [provision] **************************************************************
TASK: [servers | create droplet]
TASK: [servers | store ips]
TASK: [servers | store private ips]
TASK: [servers | create web domain]
TASK: [servers | create events domain]
TASK: [servers | create api domain]
TASK: [servers | add hosts to the inventory group]
PLAY [digital_ocean]
TASK: [Wait for port 22 to become available.]

PLAY [servers] ****************************************************************
GATHERING FACTS
TASK: [common | remove Landscape packages (cause unnecessary delay)]
TASK: [common | update apt cache once a day]
TASK: [common | install development tools]
TASK: [common | add public keys]
TASK: [common | add GitHub to known hosts]
TASK: [common | create working directory]

PLAY [database] ***************************************************************
GATHERING FACTS
TASK: [mysql | install]
TASK: [mysql | install Python interface]
TASK: [mysql | install Java interface]
TASK: [mysql | create configuration file]
TASK: [mysql | create worldskills_concrete5 database]
TASK: [mysql | create worldskills_concrete5 user]
TASK: [mysql | get worldskills_concrete5 dump from prod]
TASK: [mysql | copy worldskills_concrete5 normalize script]
TASK: [mysql | create worldskills_auth database]
TASK: [mysql | create worldskills_auth user]
TASK: [mysql | get worldskills_auth dump from prod]
TASK: [mysql | copy worldskills_auth normalize script]
TASK: [mysql | create worldskills_events database]
TASK: [mysql | create worldskills_events user]
TASK: [mysql | get worldskills_events dump from prod]
TASK: [mysql | copy worldskills_events normalize script]
TASK: [rabbitmq | import repository key]
TASK: [rabbitmq | add apt repository]
TASK: [rabbitmq | install]
TASK: [rabbitmq | enable plugins]
TASK: [rabbitmq | run service]
TASK: [rabbitmq | add users]
TASK: [rabbitmq | remove default guest user]
NOTIFIED: [mysql | restart mysql]
NOTIFIED: [mysql | import worldskills_concrete5 database]
NOTIFIED: [mysql | run worldskills_concrete5 normalize script]
NOTIFIED: [mysql | import worldskills_auth database]
NOTIFIED: [mysql | run worldskills_auth normalize script]
NOTIFIED: [mysql | import worldskills_events database]
NOTIFIED: [mysql | run worldskills_events normalize script]
NOTIFIED: [rabbitmq | restart rabbitmq]

PLAY [api] ********************************************************************
GATHERING FACTS
TASK: [tomcat | install Java]
TASK: [tomcat | install]
TASK: [tomcat | create configuration file]
TASK: [tomcat | create logging configuration]
TASK: [tomcat | run service]
TASK: [tomcat | get wsideps from prod]
TASK: [tomcat | add wsideps to classpath]
TASK: [worldskills_api_auth | get WAR from prod]
TASK: [worldskills_api_auth | deploy]
TASK: [worldskills_api_events | get WAR from prod]
TASK: [worldskills_api_events | deploy]
NOTIFIED: [tomcat | restart tomcat]

PLAY [web] ********************************************************************
GATHERING FACTS
TASK: [apache | install]
TASK: [apache | run service]
TASK: [apache | enable mod_rewrite]
TASK: [apache | add vhosts configuration]
TASK: [apache | enable vhosts]
TASK: [php | install]
TASK: [php | ensure configuration directories exist]
TASK: [php | put WorldSkills configuration file in place]
TASK: [php | download Composer]
TASK: [php | move Composer to global location]
TASK: [npm | install]
TASK: [npm | install packages]
TASK: [worldskills_concrete5 | clone project]
TASK: [worldskills_concrete5 | create shared directory]
TASK: [worldskills_concrete5 | get files directory from prod]
TASK: [worldskills_concrete5 | set release path]
TASK: [worldskills_concrete5 | export a copy of the repo]
TASK: [worldskills_concrete5 | check for existing files directory]
TASK: [worldskills_concrete5 | remove existing files directory]
TASK: [worldskills_concrete5 | link files directory]
TASK: [worldskills_concrete5 | install Composer dependencies]
TASK: [worldskills_concrete5 | create config]
TASK: [worldskills_concrete5 | create properties file]
TASK: [worldskills_concrete5 | link release to current]
TASK: [worldskills_auth | clone project]
TASK: [worldskills_auth | create shared directory]
TASK: [worldskills_auth | create logs directory]
TASK: [worldskills_auth | create log file]
TASK: [worldskills_auth | set release path]
TASK: [worldskills_auth | export a copy of the repo]
TASK: [worldskills_auth | check for existing logs directory]
TASK: [worldskills_auth | remove existing logs directory]
TASK: [worldskills_auth | link logs directory]
TASK: [worldskills_auth | install Composer dependencies]
TASK: [worldskills_auth | create config]
TASK: [worldskills_auth | link release to current]
TASK: [worldskills_events | clone project]
TASK: [worldskills_events | set release path]
TASK: [worldskills_events | export a copy of the repo]
TASK: [worldskills_events | install npm dependencies]
TASK: [worldskills_events | install bower dependencies]
TASK: [worldskills_events | create config]
TASK: [worldskills_events | link release to current]
NOTIFIED: [apache | restart apache]
NOTIFIED: [worldskills_events | build worldskills_events]

PLAY RECAP ********************************************************************
127.0.0.1                  : ok=7    changed=3    unreachable=0    failed=0
46.101.255.100             : ok=21   changed=15   unreachable=0    failed=0
46.101.255.101             : ok=54   changed=42   unreachable=0    failed=0
46.101.255.110             : ok=40   changed=34   unreachable=0    failed=0
</code></pre>
<figcaption>Output playbook</figcaption>
</figure>

<p>
After the playbook has been executed the WorldSkills website can be accessed on the proof-of-concept infrastructure at www.worldskills.ch.
</p>

<figure class="affix">
<img src="images/screenshot-proof-of-concept.png" />
<figcaption>Screenshot Proof-of-concept</figcaption>
</figure>

<p>
As the whole playbook is idempotent, it can be executed repeatedly without making any changes. Executing it without making any modifications takes 3 minutes.
</p>

<p>
All system tests are executed after the main Ansible playbook. They check content on various URLs to make sure the applications are running as expected.
</p>

<figure class="affix">
<pre><code class="small">PLAY [all] ********************************************************************
TASK: [check Events API]
TASK: [verify Events API]
TASK: [check Auth API]
TASK: [verify Auth API]
TASK: [check concrete5]
TASK: [verify concrete5]
TASK: [check Auth application]
TASK: [verify Auth application]
TASK: [check Events application]
TASK: [verify Events application]

PLAY RECAP ********************************************************************
localhost                  : ok=5    changed=0    unreachable=0    failed=0   </code></pre>
<figcaption>Output system tests</figcaption>
</figure>

<h3 class="break">Verification</h3>

<p>
The requirements outlined for the proof-of-concept are verified manually against the running proof-of-concept infrastructure.
</p>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R01 PHP applications</th>
</tr>
<tr>
<th>
Acceptance Criteria
</th>
<td>
Each PHP application is running and can be accessed with a web browser.
</td>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
The website is running at http://www.worldskills.ch/ and displays information from the Events web service. Clicking on Login redirects to http://auth.worldskills.ch/ where the Auth application is running as expected.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R02 Java applications</th>
</tr>
<tr>
<th>
Acceptance Criteria
</th>
<td>
Each Java application is running and can be accessed over HTTP/S.
</td>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
The Auth and the Events web service are running. Events are listed when accessing https://api.worldskills.ch/events and organizational groups are listed on https://api.worldskills.ch/auth/ws_entities/1.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R03 Server configuration</th>
</tr>
<tr>
<th>
Acceptance Criteria
</th>
<td>
A configuration file gets modified and the change is pushed to the code repository. The new configuration file is automatically deployed to the server and the affected applications loads the new configuration.
</td>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
After increasing the value of <code>query_cache_size</code> in the file <code>roles/mysql/templates/my.cnf.j2</code> and pushing the change to GitHub, the playbook is executed and the change is deployed to the server. The MySQL server automatically gets restarted because of the configuration file change.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R04 Application deployment</th>
</tr>
<tr>
<th>
Acceptance Criteria
</th>
<td>
A new version of an application gets pushed to the code repository. Automated tests of the application are executed and if they pass the application gets deployed to the server. The old version keeps responding to requests until the new version is ready.
</td>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
The <code>favicon.ico</code> of the main website is changed and pushed to GitHub. The unit tests are executed on Codeship and the Ansible playbook is checked out. With a separate playbook the changed file gets deployed to the server.
</p>
<p>
The new Favicon is only visible after the new version of the website is completely ready.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R05 Staging environment</th>
</tr>
<tr>
<th>
Acceptance Criteria
</th>
<td>
A new version of a web service is pushed in a separate branch, the functionality is available for testing in a staging environment within 30&nbsp;minutes from the push.
</td>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
A new branch called <em>staging-timetable</em> is created for the Events web service and pushed to GitHub. The unit tests are executed on Codeship and the Ansible playbook is checked out. A new staging environment is created on DigitalOcean and all applications are installed.
</p>
<p>
The new staging environment can be used at http://staging-timetable-www.worldskills.ch/ after 19 minutes.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R11 Configuration files</th>
</tr>
<tr>
<th>
Acceptance Criteria
</th>
<td>
All configuration files are stored in a code repository.
</td>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
Configuration files are part of the Ansible playbook and stored in the code repository.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R12 Change history</th>
</tr>
<tr>
<th>
Acceptance Criteria
</th>
<td>
Every change must be traceable by a developer. Associated with every change is an explanation.
</td>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
Every change gets commited to the code repository. An explanation is included in the commit comment.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R13 Open Source</th>
</tr>
<tr>
<th>
Acceptance Criteria
</th>
<td>
All software used for infrastructure has to be built on open-source software. This guarantees that components can easily be ported to different providers or maintainers. It also allows other Members to easily copy parts of the infrastructure.
</td>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
The playbook can be executed with Ansible which is Open Source. The required Python dependencies <a href="#ref-dopy">dopy</a> and <a href="#ref-httplib2">httplib2</a> are Open Source too.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R14 Test environment</th>
</tr>
<tr>
<th>
Acceptance Criteria
</th>
<td>
To test configuration changes to it, the whole or part of the infrastructure can be started in a test environment. This is different from the staging environment in that the test environment can be local and automated tests are executed against it.
</td>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
The playbook can be executed locally with Vagrant. Automated tests can be executed locally too.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R15 Encrypted passwords</th>
</tr>
<tr>
<th>
Acceptance Criteria
</th>
<td>
Server passwords should be stored only encrypted on third-party systems. The advantage of storing encrypted passwords in the code repository and sharing a key file instead of sharing the passwords in a file, is that the file doesn't need to be updated for everyone each time a password is added.
</td>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
Sensitive data is stored inside the Ansible Vault file.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R16 Superuser access</th>
</tr>
<tr>
<th>
Acceptance Criteria
</th>
<td>
In case of a problem that only occurs in a certain environment, a developer needs unrestricted access to the server to debug the error and try out different solutions.
</td>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
The servers on DigitalOcean can be accessed directly with SSH.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R17 Custom software</th>
</tr>
<tr>
<th>
Acceptance Criteria
</th>
<td>
New software can be installed without restrictions. New features or analytics tools might require the installation of additional software.
</td>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
Any software for Linux can be installed on the servers. As an example, <em>rsync</em> was installed on the servers to get data files for the website from the existing infrastructure.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<br/>

<figure>
<table class="full">
<colgroup>
    <col class="col-twenty" />
    <col />
</colgroup>
<tbody>
<tr>
<th>Requirement</th>
<th>R18 Learning curve</th>
</tr>
<tr>
<th>
Acceptance Criteria
</th>
<td>
How to use the software system to install and configure the infrastructure can be learned quickly so all developers can make changes to the infrastructure without spending weeks studying it.
</td>
</tr>
<tr>
<th>Verification</th>
<td>
<p>
Ansible can be understood within one day. The development of the Ansible playbook took about about 2 weeks.
</p>
</td>
</tr>
<tr>
<th>Verdict</th>
<td>
Pass <img src="images/sign-check.svg" width="16" height="16" alt="" class="icon">
</td>
</tr>
</tbody>
</table>
</figure>

<p>
The proof-of-concept passes all 13 requirements that were defined for it. The proof-of-concept covers all Must and Should requirements of the infrastructure.
</p>

</div>


<div class="chapter">

<h2 class="break">Conclusions</h2>

<h3>Ansible and Idempotence</h3>

<p>
The project has shown that idempotence on a computer system is hard. This not only applies to Ansible but to all server management software. There are always multiple sources for changes on a modern computer system and almost infinite states exist. Reducing the configuration of the whole system to one code repository (the Ansible playbook in this case) helps by reducing dependencies.
</p>

<p>
The promise of idempotence is also not true in every case with Ansible as some attributes require additional manual actions after a change. Changing the physical server location inside the <em>digital_ocean</em> module for example won't move the server from one location to another - the attribute is only used during creation of the server. Another example is the <em>copy</em> module which makes sure that a file exists in the specified location. The module checks this with every execution, however when the file path is changed (e.g. because it had a typo in it) it only makes sure the file exists in the new location. To delete the old file a separate task has to be written in Ansible. So the developers always have to be aware of what the module exactly does and they need to take care of cleanup operations by themselves.
</p>

<p>
Given that the whole infrastructure can be duplicated automatically one could think of using this for automated testing of every change to the Ansible playbook. However as the creation of such a testing environment takes around 20 minutes, every little change to the infrastructure would also take at least that long - so this option is not considered useful. Instead the choice of creating a staging environment is left up to developer if a certain change needs to be tested in more detail.
</p>

<p>
In summary the usage of Ansible makes the infrastructure more structured and changes more visible while keeping it flexible for future developments (like the integration of Docker).
</p>

<h3 class="break">Staging Environment</h3>

<p>
The time required to create the staging environment is considered a potential problem, especially as it gets worse with every additional application. The most time is used by the build of JavaScript and PHP applications. Storing compiled versions of these applications on a central server would improve the build time of a new staging environment but requires more development outside the scope of this project.
</p>

<p>
Another option of speeding up the creation of the staging environment would be to clone the server completely (on system level) instead of recreating it every time with Ansible. However this only works within the same hosting provider (not even with all hosting providers) and one of the targets of this projects was to be independent from the hosting provider. By recreating the servers each time the developers can be sure that the playbook is ready to be used with a different hosting provider.
</p>

<p>
The staging environment also suits well for more end-user tests and automated tests with tools like <a href="#ref-selenium">Selenium</a>.
</p>

<h3>Software Containers</h3>

<p>
The Evaluation showed that software containers have potential in abstracting the deployment by bundling applications into their own unit and separating the infrastructure and applications. However adopting this technology requires some changes to the applications and the deployment needs to be orchestrated with another software increasing the complexity of the whole system in total.
</p>

<p>
Applications can more easily be moved between servers and are less error prone to configuration mistakes. The advantages need to be carefully weighed in each case and testing new approaches before implementing them in production is recommended.
</p>

<h3>Objectives</h3>

<p>
The objectives of creating a versioned, testable and reproducible infrastructure where all changes are visible to the whole IT team has been fully achieved. The combination of multiple technologies and services allows quick modifications and the results are visible for everyone.
</p>

<figure class="affix">
<img src="images/before-after.svg" />
<figcaption>Comparison before and after infrastructure concept</figcaption>
</figure>

<p>
Comparing the deployment pipeline before and after the infrastructure concept shows that both infrastructure changes and application deployments follow now the same steps. Manual actions on the server and custom application deployment scripts have been replaced with Ansible.
</p>

<p>
Handling changes to the infrastructure the same way as changes to application helps the developers in their daily work as the processes and feedback loops are aligned. They get notified about modifications to the infrastructure in the chat application Slack, can review the diff on GitHub and give feedback there. The testing and deployment status is displayed in Slack as well and correction of failed changes can be organized right there with the developers online.
</p>

<h3 class="break">Recommendations</h3>

<p>
It is recommended to keep an eye on the development of software containers and re-evaluate their usage in six months. Suitable applications should then be migrated within three months if the software containers would bring reasonable advantages.
</p>

<p>
Using Ansible for provisioning and deployment of the infrastructure makes sense. It is recommended to extend the proof-of-concept playbook with all applications running on the current infrastructure. Once the setup of the new infrastructure is completed a migration to it can be done with Ansible as outlined before. The targeted time for this migration is fall 2015.
</p>

<p>
As the setup of a staging environment takes some time it's questionable how useful it is for small features. Investing into unit tests that can be executed locally is recommended here. Being able to replicate the complete infrastructure is useful for bigger features which join multiple services. It is recommended to implement the staging environment as described in the architecture documentation but use it only if needed and further invest in automated system tests.
</p>

</div>


<div class="chapter">

<h2 class="break">Appendix</h2>


<h3>Bibliography</h3>

<p id="ref-java-cookbook" class="reference-item">
    <span class="ref">Agile Orbit (2015): java Cookbook</span>.
    <a href="https://supermarket.chef.io/cookbooks/java" target="_blank">https://supermarket.chef.io/cookbooks/java</a>
    (retrieved&nbsp;21.&nbsp;April&nbsp;2015)
</p>

<p id="ref-ansible-docs" class="reference-item">
    <span class="ref">Ansible, Inc. (2015): Ansible Documentation. April 2015</span>.
    <a href="http://docs.ansible.com/" target="_blank">http://docs.ansible.com/</a>
    (retrieved&nbsp;20.&nbsp;April&nbsp;2015)
</p>

<p id="ref-ansible-tower-pricing" class="reference-item">
    <span class="ref">Ansible, Inc. (2015): Ansible Tower Pricing</span>.
    <a href="http://www.ansible.com/pricing" target="_blank">http://www.ansible.com/pricing</a>
    (retrieved&nbsp;13.&nbsp;July&nbsp;2015)
</p>

<p id="ref-ansible-tower" class="reference-item">
    <span class="ref">Ansible, Inc. (2015): Ansible Tower</span>.
    <a href="http://www.ansible.com/tower" target="_blank">http://www.ansible.com/tower</a>
    (retrieved&nbsp;18.&nbsp;June&nbsp;2015)
</p>

<p id="ref-ansible" class="reference-item">
    <span class="ref">Ansible, Inc. (2015): Ansible</span>.
    <a href="http://www.ansible.com/" target="_blank">http://www.ansible.com/</a>
    (retrieved&nbsp;20.&nbsp;April&nbsp;2015)
</p>

<p id="ref-tomcat-maven" class="reference-item">
    <span class="ref">Apache Software Foundation (2013): Apache Tomcat Maven Plugin</span>. November 2013. Version&nbsp;2.2.
    <a href="https://tomcat.apache.org/maven-plugin-2.2/" target="_blank">https://tomcat.apache.org/maven-plugin-2.2/</a>
    (retrieved&nbsp;18.&nbsp;July&nbsp;2015)
</p>

<p id="ref-apache-http" class="reference-item">
    <span class="ref">Apache Software Foundation (2015): Apache HTTP Server Project</span>.
    <a href="https://httpd.apache.org/" target="_blank">https://httpd.apache.org/</a>
    (retrieved&nbsp;19.&nbsp;July&nbsp;2015)
</p>

<p id="ref-tomcat-documentation" class="reference-item">
    <span class="ref">Apache Software Foundation (2015): Apache Tomcat 7 Configuration Reference - The Context Container</span>. June 2015. Version&nbsp;7.0.63.
    <a href="https://tomcat.apache.org/tomcat-7.0-doc/config/context.html" target="_blank">https://tomcat.apache.org/tomcat-7.0-doc/config/context.html</a>
    (retrieved&nbsp;18.&nbsp;July&nbsp;2015)
</p>

<p id="ref-yaml" class="reference-item">
    <span class="ref">Ben-Kiki O., Evans C., Döt Net I. (2009): YAML Ain’t Markup Language</span>.
    <a href="http://www.yaml.org/spec/1.2/spec.html" target="_blank">http://www.yaml.org/spec/1.2/spec.html</a>
    (retrieved&nbsp;31.&nbsp;May&nbsp;2015)
</p>

<p id="ref-cfengine" class="reference-item">
    <span class="ref">CFEngine AS (2015): CFEngine</span>.
    <a href="http://cfengine.com/" target="_blank">http://cfengine.com/</a>
    (retrieved&nbsp;13.&nbsp;July&nbsp;2015)
</p>

<p id="ref-chef" class="reference-item">
    <span class="ref">Chef Software, Inc. (2015): Chef</span>.
    <a href="https://www.chef.io/" target="_blank">https://www.chef.io/</a>
    (retrieved&nbsp;20.&nbsp;April&nbsp;2015)
</p>

<p id="ref-supermarket" class="reference-item">
    <span class="ref">Chef Software, Inc. (2015): Supermarket</span>.
    <a href="https://supermarket.chef.io/" target="_blank">https://supermarket.chef.io/</a>
    (retrieved&nbsp;21.&nbsp;April&nbsp;2015)
</p>

<p id="ref-small-n-flat" class="reference-item">
    <span class="ref">Chesne A. (2014): small-n-flat</span>.
    <a href="http://paomedia.github.io/small-n-flat/" target="_blank">http://paomedia.github.io/small-n-flat/</a>
    (retrieved&nbsp;3.&nbsp;November&nbsp;2015)
</p>

<p id="ref-codeship" class="reference-item">
    <span class="ref">Codeship (2015): Codeship</span>.
    <a href="https://codeship.com/" target="_blank">https://codeship.com/</a>
    (retrieved&nbsp;19.&nbsp;July&nbsp;2015)
</p>

<p id="ref-codeship-security" class="reference-item">
    <span class="ref">Codeship (2015): Security Guidlines of Codeship</span>.
    <a href="https://codeship.com/security" target="_blank">https://codeship.com/security</a>
    (retrieved&nbsp;4.&nbsp;June&nbsp;2015)
</p>

<p id="ref-rkt" class="reference-item">
    <span class="ref">CoreOS, Inc. (2015): rkt Documentation</span>.
    <a href="https://coreos.com/rkt/docs/0.5.6" target="_blank">https://coreos.com/rkt/docs/0.5.6</a>
    (retrieved&nbsp;13.&nbsp;July&nbsp;2015)
</p>

<p id="ref-ansible-github" class="reference-item">
    <span class="ref">DeHaan M. (2014): Ansible GitHub</span>.
    <a href="https://github.com/ansible/ansible" target="_blank">https://github.com/ansible/ansible</a>
    (retrieved&nbsp;26.&nbsp;July&nbsp;2015)
</p>

<p id="ref-ansible-github-9877" class="reference-item">
    <span class="ref">DeHaan M. (2014): Ansible GitHub issue #9877</span>.
    <a href="https://github.com/ansible/ansible/issues/9877" target="_blank">https://github.com/ansible/ansible/issues/9877</a>
    (retrieved&nbsp;24.&nbsp;June&nbsp;2015)
</p>

<p id="ref-ansible-github-999" class="reference-item">
    <span class="ref">DeHaan M. (2014): Ansible GitHub issue #999</span>.
    <a href="https://github.com/ansible/ansible-modules-core/pull/999" target="_blank">https://github.com/ansible/ansible/issues/999</a>
    (retrieved&nbsp;24.&nbsp;June&nbsp;2015)
</p>

<p id="ref-digital-ocean" class="reference-item">
    <span class="ref">DigitalOcean, Inc. (2015): Digital Ocean</span>.
    <a href="https://www.digitalocean.com/" target="_blank">https://www.digitalocean.com/</a>
    (retrieved&nbsp;26.&nbsp;July&nbsp;2015)
</p>

<p id="ref-docker" class="reference-item">
    <span class="ref">Docker, Inc. (2015): Docker</span>.
    <a href="https://www.docker.com/" target="_blank">https://www.docker.com/</a>
    (retrieved&nbsp;20.&nbsp;April&nbsp;2015)
</p>

<p id="ref-docker-docs" class="reference-item">
    <span class="ref">Docker, Inc. (2015): Docker Documentation</span>. Version&nbsp;v1.5.
    <a href="https://docs.docker.com/v1.5/" target="_blank">https://docs.docker.com/v1.5/</a>
    (retrieved&nbsp;7.&nbsp;April&nbsp;2015)
</p>

<p id="ref-docker-hub" class="reference-item">
    <span class="ref">Docker, Inc. (2015): Docker Hub</span>.
    <a href="https://hub.docker.com/" target="_blank">https://hub.docker.com/</a>
    (retrieved&nbsp;13.&nbsp;July&nbsp;2015)
</p>

<p id="ref-gpl-license" class="reference-item">
    <span class="ref">Free Software Foundation, Inc. (2007): GNU General Public License</span>. June 2007. Version&nbsp;3.
    <a href="https://www.gnu.org/licenses/gpl.html" target="_blank">https://www.gnu.org/licenses/gpl.html</a>
    (retrieved&nbsp;13.&nbsp;July&nbsp;2015)
</p>

<p id="ref-ansible-devops" class="reference-item">
    <span class="ref">Geerling J. (2015): Ansible for DevOps</span>. May 2015.
    ISBN&nbsp;978-0-9863934-0-2
</p>

<p id="ref-github" class="reference-item">
    <span class="ref">GitHub, Inc. (2015): GitHub</span>.
    <a href="https://github.com/" target="_blank">https://github.com/</a>
    (retrieved&nbsp;19.&nbsp;July&nbsp;2015)
</p>

<p id="ref-github-security" class="reference-item">
    <span class="ref">GitHub, Inc. (2015): GitHub Security</span>.
    <a href="https://help.github.com/articles/github-security/" target="_blank">https://help.github.com/articles/github-security/</a>
    (retrieved&nbsp;4.&nbsp;June&nbsp;2015)
</p>

<p id="ref-docker-php" class="reference-item">
    <span class="ref">Gravi T. (2015): Docker Official Image packaging for PHP</span>.
    <a href="https://github.com/docker-library/php" target="_blank">https://github.com/docker-library/php</a>
    (retrieved&nbsp;7.&nbsp;April&nbsp;2015)
</p>

<p id="ref-httplib2" class="reference-item">
    <span class="ref">Gregorio J. (2015): httplib2</span>.
    <a href="https://pypi.python.org/pypi/httplib2" target="_blank">https://pypi.python.org/pypi/httplib2</a>
    (retrieved&nbsp;19.&nbsp;July&nbsp;2015)
</p>

<p id="ref-vagrant" class="reference-item">
    <span class="ref">HashiCorp (2015): Vagrant</span>.
    <a href="https://www.vagrantup.com/" target="_blank">https://www.vagrantup.com/</a>
    (retrieved&nbsp;13.&nbsp;July&nbsp;2015)
</p>

<p id="ref-ansible-up-running" class="reference-item">
    <span class="ref">Hochstein L. (2015): Ansible: Up and Running</span>. March 2015. Early release revision 4.
    ISBN&nbsp;063-6-920-03562-6
</p>

<p id="ref-iso-25010" class="reference-item">
    <span class="ref">International Organization for Standardization (2011): ISO/IEC 25010:2011</span>.
    <a href="http://www.iso.org/iso/iso_catalogue/catalogue_tc/catalogue_detail.htm?csnumber=35733" target="_blank">http://www.iso.org/iso/iso_catalogue/catalogue_tc/catalogue_detail.htm?csnumber=35733</a>
    (retrieved&nbsp;13.&nbsp;July&nbsp;2015)
</p>

<p id="ref-taste-test" class="reference-item">
    <span class="ref">Jaynes M. (2014): Taste Test: Puppet, Chef, SaltStack, Ansible</span>. June 2014.
    <a href="https://devopsu.com/books/taste-test-puppet-chef-salt-stack-ansible.html" target="_blank">https://devopsu.com/books/taste-test-puppet-chef-salt-stack-ansible.html</a>
    (retrieved&nbsp;7.&nbsp;April&nbsp;2015)
</p>

<p id="ref-plesk" class="reference-item">
    <span class="ref">Parallels IP Holdings GmbH (2015): Plesk</span>.
    <a href="http://parallels.com/en/products/plesk" target="_blank">http://parallels.com/en/products/plesk</a>
    (retrieved&nbsp;7.&nbsp;April&nbsp;2015)
</p>

<p id="ref-puppet" class="reference-item">
    <span class="ref">Puppet Labs (2015): Open Source Puppet</span>.
    <a href="https://puppetlabs.com/puppet/puppet-open-source" target="_blank">https://puppetlabs.com/puppet/puppet-open-source</a>
    (retrieved&nbsp;13.&nbsp;July&nbsp;2015)
</p>

<p id="ref-pip" class="reference-item">
    <span class="ref">PyPA (2014): pip</span>.
    <a href="https://pip.pypa.io/en/stable/" target="_blank">https://pip.pypa.io/en/stable/</a>
    (retrieved&nbsp;13.&nbsp;July&nbsp;2015)
</p>

<p id="ref-rex" class="reference-item">
    <span class="ref">(R)?ex - A simple framework to simplify system administration and datacenter automation</span>.
    <a href="http://www.rexify.org/" target="_blank">http://www.rexify.org/</a>
    (retrieved&nbsp;13.&nbsp;July&nbsp;2015)
</p>

<p id="ref-saltstack" class="reference-item">
    <span class="ref">SaltStack (2015): SaltStack automation for CloudOps, ITOps &amp; DevOps at scale</span>.
    <a href="http://saltstack.com/" target="_blank">http://saltstack.com/</a>
    (retrieved&nbsp;13.&nbsp;July&nbsp;2015)
</p>

<p id="ref-apt" class="reference-item">
    <span class="ref">SPI (2015): Apt, Advanced Package Tool</span>.
    <a href="https://wiki.debian.org/Apt" target="_blank">https://wiki.debian.org/Apt</a>
    (retrieved&nbsp;13.&nbsp;July&nbsp;2015)
</p>

<p id="ref-arc42" class="reference-item">
    <span class="ref">Starke G., Hruschka P. (2014): arc42 Template</span>. March 2012. Version&nbsp;6.0.
    <a href="http://www.arc42.de/template/" target="_blank">http://www.arc42.de/template/</a>
    (retrieved&nbsp;13.&nbsp;July&nbsp;2015)
</p>

<p id="ref-rbenv" class="reference-item">
    <span class="ref">Stephenson S. (2013): rbenv</span>. July 2014.
    <a href="https://github.com/sstephenson/rbenv" target="_blank">https://github.com/sstephenson/rbenv</a>
    (retrieved&nbsp;26.&nbsp;July&nbsp;2015)
</p>

<p id="ref-selenium" class="reference-item">
    <span class="ref">Selenium Project (2015): Selenium</span>.
    <a href="http://docs.seleniumhq.org/" target="_blank">http://docs.seleniumhq.org/</a>
    (retrieved&nbsp;19.&nbsp;July&nbsp;2015)
</p>

<p id="ref-slack" class="reference-item">
    <span class="ref">Slack Technologies, Inc. (2015): Slack</span>.
    <a href="https://slack.com/" target="_blank">https://slack.com/</a>
    (retrieved&nbsp;19.&nbsp;July&nbsp;2015)
</p>

<p id="ref-apache-license" class="reference-item">
    <span class="ref">The Apache Software Foundation (2004): Apache License</span>. January 2004. Version&nbsp;2.0.
    <a href="https://www.apache.org/licenses/LICENSE-2.0.html" target="_blank">https://www.apache.org/licenses/LICENSE-2.0.html</a>
    (retrieved&nbsp;13.&nbsp;July&nbsp;2015)
</p>

<p id="ref-pear" class="reference-item">
    <span class="ref">The PHP Group (2015): PEAR</span>.
    <a href="http://pear.php.net/" target="_blank">http://pear.php.net/</a>
    (retrieved&nbsp;13.&nbsp;July&nbsp;2015)
</p>

<p id="ref-php-versions" class="reference-item">
    <span class="ref">The PHP Group (2015): PHP Supported Versions</span>.
    <a href="https://php.net/supported-versions.php" target="_blank">https://php.net/supported-versions.php</a>
    (retrieved&nbsp;7.&nbsp;April&nbsp;2015)
</p>

<p id="ref-docker-book" class="reference-item">
    <span class="ref">Turnbull J. (2015): The Docker Book</span>. February 2015. Version&nbsp;v1.5.0.
    ISBN&nbsp;978-0-9888202-3-4
</p>

<p id="ref-apache2-cookbook" class="reference-item">
    <span class="ref">Van Zoest S. (2015): apache2 Cookbook</span>.
    <a href="https://supermarket.chef.io/cookbooks/apache2" target="_blank">https://supermarket.chef.io/cookbooks/apache2</a>
    (retrieved&nbsp;21.&nbsp;April&nbsp;2015)
</p>

<p id="ref-cooking-chef" class="reference-item">
    <span class="ref">Vasiliev A. (2014): Cooking Infrastructure by Chef</span>.
    <a href="http://chef.leopard.in.ua/" target="_blank">http://chef.leopard.in.ua/</a>
    (retrieved&nbsp;18.&nbsp;April&nbsp;2015)
</p>

<p id="ref-infoworld-comparison" class="reference-item">
    <span class="ref">Venezia P. (2013): Puppet vs. Chef vs. Ansible vs. Salt</span>.
    <a href="http://www.infoworld.com/article/2609482/data-center/data-center-review-puppet-vs-chef-vs-ansible-vs-salt.html" target="_blank">http://www.infoworld.com/article/2609482/data-center/data-center-review-puppet-vs-chef-vs-ansible-vs-salt.html</a>
    (retrieved&nbsp;13.&nbsp;July&nbsp;2015)
</p>

<p id="ref-dopy" class="reference-item">
    <span class="ref">Viallet V. (2015): dopy</span>.
    <a href="https://pypi.python.org/pypi/dopy" target="_blank">https://pypi.python.org/pypi/dopy</a>
    (retrieved&nbsp;19.&nbsp;July&nbsp;2015)
</p>

<p id="ref-prince" class="reference-item">
    <span class="ref">YesLogic Pty. Ltd. (2015): Prince</span>.
    <a href="http://www.princexml.com/" target="_blank">http://www.princexml.com/</a>
    (retrieved&nbsp;25.&nbsp;July&nbsp;2015)
</p>

<p id="ref-yum" class="reference-item">
    <span class="ref">Yum Package Manager</span>.
    <a href="http://yum.baseurl.org/" target="_blank">http://yum.baseurl.org/</a>
    (retrieved&nbsp;13.&nbsp;July&nbsp;2015)
</p>

<p id="ref-rules-bachelor-thesis" class="reference-item">
    <span class="ref">Zürcher Hochschule für Angewandte Wissenschaften (2014): Reglement Bachelorarbeit Studiengang Informatik der ZHAW am Standort Zürich</span>. March 2015. Version&nbsp;3.3.
    <a href="https://ebs.zhaw.ch/files/documents/informatik/Reglemente/Bachelor/Bachelorarbeit/a_Reglement-Bachelorarbeit_Studiengang-Informatik_V3.3.pdf" target="_blank">https://ebs.zhaw.ch/files/documents/informatik/Reglemente/Bachelor/Bachelorarbeit/a_Reglement-Bachelorarbeit_Studiengang-Informatik_V3.3.pdf</a>
    (retrieved&nbsp;25.&nbsp;July&nbsp;2015)
</p>


<h3 class="break">List of Figures</h3>

<ul id="figures"></ul>


<h3 class="break">Software</h3>

<p>
To make the results of this thesis reproducible, the following list shows the version of the software used on the development machine:
</p>

<ul>
<li>Mac OS X 10.10.2 Build 14C1514</li>
<li>Git 1.9.5</li>
<li>Boot2Docker 1.5.0 (Docker v1.5.0, Linux v3.18.5)</li>
<li>Docker Compose 1.1.0</li>
<li>VirtualBox 4.3.26 r98988</li>
<li>Vagrant 1.7.2</li>
<li>ChefDK 0.4.0</li>
<li>Chef 12.0.3</li>
<li>Berkshelf 3.2.3</li>
<li>Tower 2.2.0 Build 274</li>
<li>Chocolat 3.1.4</li>
<li>Prince 9.0 rev 5</li>
<li>RubyGems.org 2.4.4</li>
<li>Ansible 1.9.0.1</li>
<li>Python 2.7.9</li>
<li>pip 6.0.6</li>
<li>dopy 0.3.6</li>
<li>httplib2 0.9.1</li>
<li>Atom 0.211.0</li>
<li>Slack 1.0.5</li>
</ul>

<p>
The following versions of software were used on the server:
</p>

<ul>
<li>Ubuntu 14.04</li>
<li>Apache 2.4.7</li>
<li>PHP 5.5.9</li>
<li>MySQL 5.5.43</li>
<li>RabbitMQ 3.5.3</li>
<li>Tomcat 7.0.52.0</li>
<li>OpenJDK 7u51</li>
<li>npm 1.3.10</li>
<li>Grunt 0.4.5</li>
<li>Bower 1.4.1</li>
</ul>

<h3 class="break">Glossary</h3>

<dl>
<dt>Branch</dt>
<dd>A method for making parallel code changes in a code repository.</dd>
<dt>CI</dt>
<dd>Continuous integration: Automated testing and deployment of code changes.</dd>
<dt>Cloud</dt>
<dd>Running dynamic computer software without detailed knowledge of the physical hardware.</dd>
<dt>CMS</dt>
<dd>Content management system: Software to edit websites.</dd>
<dt>Code repository</dt>
<dd>Used to store and manage source code in a version control system.</dd>
<dt>Competition</dt>
<dd>WorldSkills Competition: Biennial skills competition for youth.</dd>
<dt>CSS</dt>
<dd>Cascading Style Sheets: Language used to style websites</dd>
<dt>Database fixtures</dt>
<dd>Initial data for a database.</dd>
<dt>Debian</dt>
<dd>Variation of Linux maintained by a group of volunteers.</dd>
<dt>Deployment</dt>
<dd>Installing or updating a software on a server.</dd>
<dt>DNS</dt>
<dd>Domain Name System: Software used to resolve domain names.</dd>
<dt>Environment</dt>
<dd>Logical group of servers.</dd>
<dt>Favicon</dt>
<dd>Small icon used by web browsers to represent a website.</dd>
<dt>Git</dt>
<dd>A distributed version control system.</dd>
<dt>Horizontal scaling</dt>
<dd>Extending the capacity of a service by distributing the load over multiple servers.</dd>
<dt>HTML</dt>
<dd>Hypertext Markup Language: A markup language used on the world wide web.</dd>
<dt>HTTP</dt>
<dd>Hypertext Transfer Protocol: A protocol for transfering data of the world wide web.</dd>
<dt>HTTPS</dt>
<dd>HTTP Secure: Protocol for using HTTP in a secure way with an added encryption layer.</dd>
<dt>Idempotence</dt>
<dd>Operation that can be executed multiple times without changing the state after an initial change.</dd>
<dt>IP address</dt>
<dd>Numerical identifier of a computer within a network.</dd>
<dt>IT</dt>
<dd>Information technology: Management of data using computers.</dd>
<dt>Java</dt>
<dd>Programming language used for different purposes.</dd>
<dt>JavaScript</dt>
<dd>Programming language which can be executed in a web browser.</dd>
<dt>Kernel</dt>
<dd>Component of an operation system responsible for executing system commands.</dd>
<dt>Linux</dt>
<dd>Open Source operating system.</dd>
<dt>Mac OS X</dt>
<dd>An operating system by Apple Inc.</dd>
<dt>Maven</dt>
<dd>Dependency and build automation software for Java applications.</dd>
<dt>Member</dt>
<dd>Member organization of WorldSkills International.</dd>
<dt>mod_php</dt>
<dd>Apache web server module for running PHP.</dd>
<dt>MySQL</dt>
<dd>A relational database management system.</dd>
<dt>MySQL Query Cache</dt>
<dd>A cache of MySQL used to reduce the time required to parse the query.</dd>
<dt>OS</dt>
<dd>Operating system: Software running on a computer and providing the possibility to run other software.</dd>
<dt>OSS</dt>
<dd>Open source software: Software where the source code is licensed in a way that it can be viewed and changed.</dd>
<dt>PDF</dt>
<dd>Portable Document Format: A portable file format for documents.</dd>
<dt>PHP</dt>
<dd>A programming language used mostly on the world wide web.</dd>
<dt>PHP-FPM</dt>
<dd>PHP FastCGI Process Manager: An way to run PHP so a webserver can execute PHP applications.</dd>
<dt>Python</dt>
<dd>Dynamic programming language used for various purposes.</dd>
<dt>RabbitMQ</dt>
<dd>Open source message queue software.</dd>
<dt>Race condition</dt>
<dd>Unintended dependency of the ouput of a system on time.</dd>
<dt>Ruby</dt>
<dd>A dynamic programming language used on the web and on the command line.</dd>
<dt>Secretariat</dt>
<dd>People working for the organization WorldSkills International.</dd>
<dt>Servlet container</dt>
<dd>An application to run Java web applications.</dd>
<dt>SSH</dt>
<dd>Secure Shell: An encrypted protocol for executing commands on a remote server.</dd>
<dt>Staging</dt>
<dd>A dedicated environment to test software applications.</dd>
<dt>Superuser</dt>
<dd>Special user account on a system with unrestricted access.</dd>
<dt>SUSE Linux</dt>
<dd>Variation of Linux supported by Novell.</dd>
<dt>Symlink</dt>
<dd>Symbolic link: A file or directory pointing to another path.</dd>
<dt>Template file</dt>
<dd>File with placeholders that gets compiled into its final form by replacing those placeholders with variable values.</dd>
<dt>Tomcat</dt>
<dd>A web server to run Java applications.</dd>
<dt>Ubuntu</dt>
<dd>Variation of Linux maintained by Canonical Ltd.</dd>
<dt>URL</dt>
<dd>Uniform resource locator: Reference to a document in a network.</dd>
<dt>Vagrant</dt>
<dd>Software for running virtual machines locally for development.</dd>
<dt>Virtual machine</dt>
<dd>Emulation of a computer system used to run multiple machines on the same hardware.</dd>
<dt>Webhook</dt>
<dd>A URL that gets called by an online service after a specific action to let another service know about the outcome of the action.</dd>
<dt>Web service</dt>
<dd>Interface to provide programmatic access to data and functionality over the web.</dd>
<dt>YAML</dt>
<dd>YAML Ain't Markup Language: File format for writting structured data.</dd>
</dl>

</div>


<footer>&#160;</footer>


</body>
</html>