source commit: e645b13

06-containers-on-the-cloud.md

@@ -0,0 +1,94 @@
+---
+title: Creating Containers on the Cloud
+teaching: 20
+exercises: 0
+questions: How can I create Docker containers in the cloud?
+objectives:
+- Demonstrate how to effect creation of a container from the Docker image in the cloud.
+- Gain an initial experience of the container functionality provided by the Bitbucket
+ repository storage service.
+keypoints:
+- You can create Docker containers on cloud computing resources just using a web browser.
+- Bitbucket is an online repository storage service that can create Docker containers
+ to perform computation in response to files changing in your repositories.
+---
+
+## Containers can be created on cloud computing platforms
+
+There are lots of ways containers can be created on cloud computing platforms (a.k.a., "in the cloud"). Most commercial cloud providers now offer a container hosting service that will connect to the Docker Hub in order to fetch the container images that you specify, and charge for the time and resources that the containers use. The container hosting will usually be significantly cheaper than full virtualisation services that might be on offer.
+
+Note also that most cloud providers will give you sign-up credit that you can use for free after you first create your account.
+
+For this lesson, though, we instead use an excellent software project repository platform, Bitbucket, that allows users a monthly quota of minutes for which containers of your choice can be run. Bitbucket allows containers to be created in response to the modification of files within your software project.
+
+::::::::::::::::::::::::::::::::::::::::: callout
+
+## There are many excellent sites for storing public software repositories
+
+- Note that Bitbucket, GitHub and GitLab all achieve similar functions.
+- Bitbucket offers container-based features that are easier to get at than the equivalent functions in GitHub, although GitHub will soon catch up when they release their GitHub Actions functionality publicly.
+
+
+::::::::::::::::::::::::::::::::::::::::::::::::::
+
+### Running a container in the cloud, using your Bitbucket account
+
+Because the ability to use the `git` version management tool is not a prerequisite of this session, we will use Bitbucket in an atypical manner. However we should still be able to clearly see Bitbucket's cloud servers running a container of your choice, under your control.
+
+- Open a web browser window and visit [https://bitbucket.org/](https://bitbucket.org/).
+- Log into your Bitbucket account.
+- Click the "create" (or just "+", if the menu is not expanded) button near the top-left of the page.
+- In the Create menu that appears, choose "Repository".
+- You will need to fill in the "Create new repository" form:
+ - You need to choose a Repository name: I am going to choose "use-my-container", and you are welcome to do the same.
+ - I unchecked the "This is a private repository" button, to make my repository public
+ - All of the other defaults should be OK, including the advanced settings
+- Activate the "Create repository" button
+- A page with heading "Let's put some bits in your bucket" appears, since the aim of Bitbucket is to host repositories of code and data... but we will ignore this workflow, and instead, you should click "Pipelines" in the second menu in from the left.
+- A page that promotes Bitbucket Pipelines should appear. This page also notes that you have 500 free minutes per month (for a free account): this is minutes of time that your containers are allowed to run, on Bitbucket's cloud servers. You just need to click the "Start using Pipelines" link in the bottom-centre.
+- Under the "Choose a language template" heading that appears, choose the "Other" pull-down menu, and select "Other" from the bottom of that list.
+- If an example "bitbucket-pipelines.yml" file appears, then all is well, and you can continue with the next section.
+
+### Edit your repository's `bitbucket-pipelines.yml` file through the web
+
+You should be looking at a web-based text editor that is headed "bitbucket-pipelines.yml". The one that I see has 13 lines, all of which are numbered on the left of the text editor.
+
+Bitbucket Pipelines allow you to specify software tools to run, for example, in response to files being changed in your Bitbucket projects. The Bitbucket servers run your software tools within Docker containers, and thus Bitbucket Pipelines can specify Docker images to fetch from the Docker Hub.
+
+Change your repository's `bitbucket-pipelines.yml` file to be similar to the following example, but note that you need to replace the Docker Hub user ID (alice in example) with yours. Also, ensure that your indentation steps in line-by-line, the language being used (YAML) gives significance to the indentation of the lines.
+
+```
+image: alice/my-container
+
+pipelines:
+ default:
+ - step:
+ script:
+ - /bin/cat /root/my_message
+```
+
+Click the "commit file" button. After you commit your `bitbucket-pipelines.yml` file, the Bitbucket Pipeline will download the Docker image you specified from the Docker Hub, and display the progress of the computations it runs.
+
+When using the `docker run` command (as you have done previously), the container takes some default actions after being created, which are specified in your Dockerfile (e.g., the `CMD` line). Bitbucket Pipelines disable these default actions, instead using the commands listed under the "script:" section in your `bitbucket-pipelines.yml`. Note that hyphens at the same indentation level are treated as an itemised list. There is only one item in our `script:` list, namely the command `/bin/cat /root/my_message`.
+
+If the pipeline runs successfully, a green heading containing a tick icon will be shown near the top of the page. On the right-hand-side of the page, you should see the following headings:
+
+- Build step
+- `/bin/cat /root/my_message`
+- Build teardown
+
+Click on the `/bin/cat /root/my_message` heading, and you should see that your custom message was shown.
+
+While it is difficult to argue that this container achieves important computational work, you have, nonetheless, demonstrated that Docker images that you create can be run on the cloud. Moreover, many cloud organisations that are willing to create containers from your images will offer generous allowances to you to do so, even if you only have a free account.
+
+<!-- Going further section
+
+## Digital Ocean hosting
+
+Cloud providers such as Digital Ocean -->
+
+<!-- LocalWords: keypoints links.md endcomment Dockerfile
+ -->
+
+<!-- LocalWords: bitbucket-pipelines.yml
+ -->

08-orchestration.md

@@ -0,0 +1,59 @@
+---
+title: Container Orchestration
+teaching: 10
+exercises: 0
+questions: How can I deploy multiple containers?
+objectives: Become aware of container orchestration services.
+keypoints: Docker Compose, Kubernetes, and Docker Swarm are tools that can deploy
+ multiple containers.
+---
+
+## Container Orchestration
+
+Although you can certainly manage research workflows that use multiple containers manually, there are a number of
+container orchestration tools that you may find useful when managing workflows that use multiple containers.
+We won't go in depth on using these tools in this lesson but instead briefly describe
+a few options and point to useful resources on using these tools to allow you to explore them yourself.
+
+- Docker Compose
+- Kubernetes
+- Docker Swarm
+
+::::::::::::::::::::::::::::::::::::::::: callout
+
+## The Wild West
+
+Use of container orchestration tools for research workflows is a relatively new concept and so there
+is not a huge amount of documentation and experience out there at the moment. You may need to search
+around for useful information or, better still, contact your
+[friendly neighbourhood RSE](https://society-rse.org/community/rse-groups/) to discuss what you want to do.
+
+
+::::::::::::::::::::::::::::::::::::::::::::::::::
+
+**Docker Compose** provides a way of constructing a unified workflow (or service) made up of multiple
+individual Docker containers. In addition to the individual Dockerfiles for each container, you provide
+a higher-level configuration file which describes the different containers and how they link together
+along with shared storage definitions between the containers. Once this high-level configuration has been
+defined, you can use single commands to start and stop the orchestrated set of containers.
+
+- [Using Docker Compose for the Simple Deployment of an Integrated Drug Target Screening Platform](https://www.degruyter.com/view/journals/jib/14/2/article-20170016.xml)
+- [Docker Compose Overview](https://docs.docker.com/compose/)
+
+**Kubernetes** is an open source framework that provides similar functionality to Docker Compose. Its
+particular strengths are that is platform independent and can be used with many different container
+technologies and that it is widely available on cloud platforms so once you have implemented your workflow
+in Kubernetes it can be deployed in different locations as required. It has become the de facto standard
+for container orchestration.
+
+- [What is Kubernetes](https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/)
+
+**Docker Swarm** provides a way to scale out to multiple copies of similar containers. This potentially
+allows you to parallelise and scale out your research workflow so that you can run multiple copies and
+increase throughput. This would allow you, for example, to take advantage of multiple cores on a local
+system or run your workflow in the cloud to access more resources. Docker Swarm uses the concept of
+a manager container and worker containers to implement this distribution.
+
+- [Docker Swarm Overview](https://docs.docker.com/engine/swarm/)
+
+

CODE_OF_CONDUCT.md

@@ -0,0 +1,13 @@
+---
+title: "Contributor Code of Conduct"
+---
+
+As contributors and maintainers of this project,
+we pledge to follow the [The Carpentries Code of Conduct][coc].
+
+Instances of abusive, harassing, or otherwise unacceptable behavior
+may be reported by following our [reporting guidelines][coc-reporting].
+
+
+[coc-reporting]: https://docs.carpentries.org/topic_folders/policies/incident-reporting.html
+[coc]: https://docs.carpentries.org/topic_folders/policies/code-of-conduct.html

LICENSE.md

@@ -0,0 +1,79 @@
+---
+title: "Licenses"
+---
+
+## Instructional Material
+
+All Carpentries (Software Carpentry, Data Carpentry, and Library Carpentry)
+instructional material is made available under the [Creative Commons
+Attribution license][cc-by-human]. The following is a human-readable summary of
+(and not a substitute for) the [full legal text of the CC BY 4.0
+license][cc-by-legal].
+
+You are free:
+
+- to **Share**---copy and redistribute the material in any medium or format
+- to **Adapt**---remix, transform, and build upon the material
+
+for any purpose, even commercially.
+
+The licensor cannot revoke these freedoms as long as you follow the license
+terms.
+
+Under the following terms:
+
+- **Attribution**---You must give appropriate credit (mentioning that your work
+ is derived from work that is Copyright (c) The Carpentries and, where
+ practical, linking to <https://carpentries.org/>), provide a [link to the
+ license][cc-by-human], and indicate if changes were made. You may do so in
+ any reasonable manner, but not in any way that suggests the licensor endorses
+ you or your use.
+
+- **No additional restrictions**---You may not apply legal terms or
+ technological measures that legally restrict others from doing anything the
+ license permits. With the understanding that:
+
+Notices:
+
+* You do not have to comply with the license for elements of the material in
+ the public domain or where your use is permitted by an applicable exception
+ or limitation.
+* No warranties are given. The license may not give you all of the permissions
+ necessary for your intended use. For example, other rights such as publicity,
+ privacy, or moral rights may limit how you use the material.
+
+## Software
+
+Except where otherwise noted, the example programs and other software provided
+by The Carpentries are made available under the [OSI][osi]-approved [MIT
+license][mit-license].
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of
+this software and associated documentation files (the "Software"), to deal in
+the Software without restriction, including without limitation the rights to
+use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
+of the Software, and to permit persons to whom the Software is furnished to do
+so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+## Trademark
+
+"The Carpentries", "Software Carpentry", "Data Carpentry", and "Library
+Carpentry" and their respective logos are registered trademarks of [Community
+Initiatives][ci].
+
+[cc-by-human]: https://creativecommons.org/licenses/by/4.0/
+[cc-by-legal]: https://creativecommons.org/licenses/by/4.0/legalcode
+[mit-license]: https://opensource.org/licenses/mit-license.html
+[ci]: https://communityin.org/
+[osi]: https://opensource.org

about.md

@@ -0,0 +1,8 @@
+---
+title: About
+---
+
+{% include carpentries.html %}
+
+
+