Merge pull request #172 from singularityhub/add/builders

Adding Google Cloud Build + Storage Plugin

.gitignore

@@ -10,6 +10,7 @@ _site
 
 # Other #
 #########
+*.json
 migrations/
 make_video.sh
 celerybeat-schedule.db

CHANGELOG.md

@@ -12,6 +12,7 @@ represented by the pull requests that fixed them. Critical items to know are:
 
 
 ## [master](https://github.com/singularityhub/sregistry/tree/master) (master)
+ - Addition of Google Cloud Build, versioning, tags to collections (1.1.0)
  - adding BitBucket authentication backend
  - updating sregistry-cli to 0.0.97, catching OSError earlier
  - updating sregistry-cli to 0.0.96, and Singularity download url to use sylabs organization

Dockerfile

@@ -33,11 +33,6 @@ RUN apt-get update && apt-get install -y \
     squashfs-tools \
     build-essential
 
-# Install Singularity
-RUN git clone -b vault/release-2.6 https://www.github.com/sylabs/singularity.git
-WORKDIR singularity
-RUN ./autogen.sh && ./configure --prefix=/usr/local && make && make install
-
 # Install Python requirements out of /tmp so not triggered if other contents of /code change
 ADD requirements.txt /tmp/requirements.txt
 RUN pip install --upgrade pip
@@ -47,7 +42,7 @@ ADD . /code/
 
 ################################################################################
 # PLUGINS
-# You are free to comment out those plugins that you don't want to use
+# You are free to uncomment the plugins that you want to use
 
 # Install LDAP (uncomment if wanted)
 # RUN pip install python3-ldap
@@ -56,6 +51,9 @@ ADD . /code/
 # Install PAM Authentication (uncomment if wanted)
 # RUN pip install django-pam
 
+# Ensure Google Build Installed
+# RUN pip install sregistry[google-build]
+
 # Install Globus (uncomment if wanted)
 # RUN /bin/bash /code/scripts/globus/globus-install.sh
 

README.md

@@ -16,9 +16,10 @@ to work together using [docker-compose.yml](docker-compose.yml).
 The images are the following:
 
  - **vanessa/sregistry**: is the main uwsgi application, which serves a Django (python-based) application.
- - **nginx**: pronounced (engine-X) is the webserver. The starter application is configured for http, however you should follow the instructions to set up https properly.
- - **worker**: is the same uwsgi image, but with a running command that is specialized to perform tasks. The tasks are run via [celery](http://www.celeryproject.org/), a distributed job queue that fits nicely into Django. The celery worker uses a
+ - **nginx**: pronounced (engine-X) is the webserver. The starter application is configured for http, however you should follow the instructions to set up https properly. Note that we build a custom nginx image that takes advantage of the [nginx upload module](https://www.nginx.com/resources/wiki/modules/upload/).
+ - **worker**: is the same uwsgi image, but with a running command that is specialized to perform tasks. The tasks are run via [django-rq](https://github.com/rq/django-rq) that uses a
  - **redis**: database to organize the jobs themselves.
+ - **scheduler** jobs can be scheduled using the scheduler.
 
 For more information about Singularity Registry Server, please reference the 
 [docs](https://singularityhub.github.io/sregistry). If you have any issues, 

VERSION

@@ -0,0 +1 @@
+1.1.0

docker-compose.yml

@@ -32,13 +32,21 @@ nginx:
 redis:
   restart: always
   image: redis:latest
-  ports:
-    - "6379:6379"
 
+scheduler:
+  image: vanessa/sregistry
+  command: python /code/manage.py rqscheduler
+  volumes:
+    - .:/code
+  volumes_from:
+    - uwsgi
+  links:
+    - redis
+    - db
 
 worker:
   image: vanessa/sregistry
-  command: celery worker -A shub.celery -Q default -n default@%h -B
+  command: python /code/manage.py rqworker default
   volumes:
     - .:/code
   volumes_from:

docs/pages/client.md

@@ -6,15 +6,29 @@ permalink: /client
 toc: false
 ---
 
-<script src="assets/js/asciinema-player.js"></script>
-<link rel="stylesheet" href="assets/css/asciinema-player.css"/>
+## Singularity Pull
+
+Singularity Registry Server implements a basic version of the Sylabs Library API, 
+meaning that you can pull a container with Singularity directly. For example,
+let's say that I have a collection with a container called `collection/container:tag`.
+and my registry is served at `containers.page`. I could pull it as follows:
+
+```bash
+$ singularity pull --library https://containers.page collection/container:tag
+```
+
+# SRegistry Client
+
+Singularity Registry Global Client, or [sregistry-cli](https://github.com/singularityhub/sregistry-cli),
+is a general client to interact with Singularity images at remote endpoints, and it provides
+such an endpoint for Singularity Registry Server. We will provide
+basic instructions here, and for the full documentation, please see the [getting started guide here](https://singularityhub.github.io/sregistry-cli/client-registry). Note that you will need to [export your credentials](https://singularityhub.github.io/sregistry/credentials) in order to have authenticated interaction with sregistry.
 
-The original Singularity Registry Client was provided by [Singularity Python](https://github.com/singularityware/singularity-python), however we have moved the client to have its own module under [sregistry-cli](https://github.com/singularityhub/sregistry-cli). We recommend that you use the latter, and ask for features or updates when necessary. For the new version, see the [getting started guide here](https://singularityhub.github.io/sregistry-cli/client-registry). Note that you will need to [export your credentials](https://singularityhub.github.io/sregistry/credentials) in order to have authenticated
-interaction with sregistry.
 
 ## Install
 
 ### sregistry Installation
+
 `sregistry` is the client for Singularity Registry server. To install, you can do the following:
 
 ```

docs/pages/install/builders.md

@@ -0,0 +1,18 @@
+---
+layout: default
+title: Custom Builders and Storage
+pdf: true
+permalink: /install-builders
+toc: false
+---
+
+Currently, we support custom installation of the following builder and storage pairs. Each of these is provided
+as a plugin, so you can enable them in the same way. Instructions are included with the links below.
+
+ - [Google Cloud Build + Storage]({{ site.baseurl }}/plugin-google-build)
+
+Don't forget to go back to the [install docs](https://singularityhub.github.io/sregistry/install-server#storage) where you left off.
+
+<div>
+    <a href="/sregistry/install"><button class="previous-button btn btn-primary"><i class="fa fa-chevron-left"></i> </button></a>
+</div><br>

docs/pages/install/containers.md

@@ -8,12 +8,13 @@ toc: true
 
 # Installation: Start Containers
 
-Whether you build or not, the compose command will bring up the application (and download containers provided on Docker Hub, if they aren't in your cache).
+Whether you build or not, the compose command will bring up the application 
+(and download containers provided on Docker Hub, if they aren't in your cache).
 
 ## What containers are provided?
 
-Singularity Registy Server uses the following images, all provided on Docker Hub (or you can build the registry-specific ones
-locally):
+Singularity Registy Server uses the following images, all provided on Docker Hub 
+(or you can build the registry-specific ones locally):
 
  - [vanessa/sregistry](https://hub.docker.com/r/vanessa/sregistry): is the core application image, generated from the Dockerfile in the base of the repository.
  - [vanessa/sregistry_nginx](https://hub.docker.com/r/vanessa/sregistry_nginx/): Is the nginx container installed with the nginx upload module, intended for use with speedy uploads. It is generated from the subfolder "nginx" in the repository.
@@ -23,17 +24,24 @@ To use these images provided, you can bring up the containers like so:
 ## Start Containers
 
 ```bash
-docker-compose up -d
+$ docker-compose up -d
 ```
 
-The `-d` means detached, and that you won't see any output (or errors) to the console. You can easily restart and stop containers, either specifying the container name(s) or leaving blank to apply to all containers. Note that these commands must be run in the folder with the `docker-compose.yml`:
+The `-d` means detached, and that you won't see any output (or errors) to the 
+console. You can easily restart and stop containers, either specifying the 
+container name(s) or leaving blank to apply to all containers. Note that these 
+commands must be run in the folder with the `docker-compose.yml`:
 
 ```bash
-docker-compose restart uwsgi worker nginx
-docker-compose stop
+$ docker-compose restart uwsgi worker nginx
+$ docker-compose stop
 ```
 
-When you do `docker-compose up -d` the application should be available at `http://127.0.0.1/`, and if you've configured https, `https://127.0.0.1/`. If you need to shell into the application, for example to debug with `python manage.py shell` you can get the container id with `docker ps` and then do:
+When you do `docker-compose up -d` the application should be available at 
+`http://127.0.0.1/`, and if you've configured https, `https://127.0.0.1/`. 
+If you need to shell into the application, for example to debug with 
+`python manage.py shell` you can get the container id with `docker ps` 
+and then do:
 
 ```bash
 NAME=$(docker ps -aqf "name=sregistry_uwsgi_1")

docs/pages/install/server.md

@@ -7,9 +7,11 @@ toc: true
 ---
 
 # Installation: Web Server and Storage
+
 Before doing `docker-compose up -d` to start the containers, there are some specific things that need to be set up.
 
 ## Nginx
+
 This section is mostly for your FYI. The nginx container that we use is a custom compiled
 nginx that includes the [nginx uploads module](https://www.nginx.com/resources/wiki/modules/upload/).
 This allows us to define a server block that will accept multipart form data directly, and 
@@ -60,20 +62,42 @@ nginx:
 the image will be built from the `nginx` folder provided in the repository.
 
 ## Under Maintenance Page
+
 If it's ever the case that the Docker images need to be brought down for maintenance, a static fallback page should be available to notify the user. If you noticed in the [prepare_instance.sh](https://github.com/singularityhub/sregistry/blob/master/scripts/prepare_instance.sh) script, one of the things we installed is nginx (on the instance). This is because we need to use it to get proper certificates for our domain (for https). Before you do this, you might want to copy the index that we've provided to replace the default (some lame page that says welcome to Nginx!) to one that you can show when the server is undergoing maintainance.
 
 ```bash
 cp $INSTALL_ROOT/sregistry/scripts/nginx-index.html /var/www/html/index.html
 rm /var/www/html/index.nginx-debian.html
 ```
 
+If you want your page to use the same SSL certificates, a nginx-default.conf is also
+provided that will point to the same certificates on the server (generation discussed later):
+
+```bash
+cp $INSTALL_ROOT/sregistry/scripts/nginx-default.conf /etc/nginx/conf.d/default.conf
+```
+
 If you don't care about user experience during updates and server downtime, you can just ignore this.
 
+## Custom Domain
+
+In the [config settings file](https://github.com/singularityhub/sregistry/blob/master/shub/settings/config.py#L30)
+you'll find a section for domain names, and other metadata about your registry. You will need to update
+this to be a custom hostname that you use, and custom names and unique resource identifiers for your
+registry. For example, if you have a Google Domain and are using Google Cloud, you should be able to set it up using [Cloud DNS](https://console.cloud.google.com/net-services/dns/api/enable?nextPath=%2Fzones&project=singularity-static-registry&authuser=1). Usually this means
+creating a zone for your instance, adding a Google Domain, and copying the DNS records for
+the domain into Google Domains. Sometimes it can take a few days for changes to propogate.
+We will discuss setting up https in a later section.
+
 ## Storage
-The containers that you upload to your registry will be stored "inside" the Docker container, specifically at the location `/var/www/images`. By default, we map this location to the host in the base directory of `sregistry` in a folder called `images`. Equally, we map static web files to a folder named `static`. If you look in the [docker-compose.yml](https://github.com/singularityhub/sregistry/blob/master/docker-compose.yml) that looks something like this:
 
+By default, the containers that you upload to your registry will be stored "inside" the Docker container, specifically at the location `/var/www/images`. While it would not be reasonable to upload to Singularity Registry and then to a custom Storage, we have recently added
+[custom builders]({{ site.url }}/install-builders) that can be used to push a recipe to Singularity Registry Server, and then trigger a cloud build that will be saved in some matching cloud storage.
 
-```
+If you choose the file system default storage, we map this location to the host in the base directory of `sregistry` in a folder called `images`. Equally, we map static web files to a folder named `static`. If you look in the [docker-compose.yml](https://github.com/singularityhub/sregistry/blob/master/docker-compose.yml) that looks something like this:
+
+
+```yaml
     - ./static:/var/www/static
     - ./images:/var/www/images
 ```
@@ -87,7 +111,17 @@ Thus, you are free to test different configurations of mounting this folder. If
 
 
 ## SSL
-Getting https certificates is really annoying, and getting `dhparams.pem` takes forever. But after the domain is obtained, it's important to do. Again, remember that we are working on the host, and we have an nginx server running. You should follow the instructions (and I do this manually) in [generate_cert.sh](../scripts/generate_cert.sh). It basically comes down to:
+
+Getting https certificates is really annoying, and getting `dhparams.pem` takes forever. But after the domain is obtained, it's important to do. Again, remember that we are working on the host, and we have an nginx server running. You should follow the instructions (and I do this manually) in [generate_cert.sh](https://github.com/singularityhub/sregistry/blob/master/scripts/generate_cert.sh). 
+
+ - starting nginx
+ - installing certbot
+ - generating certificates
+ - linking them to where the docker-compose expects them
+ - add a reminder or some other method to renew within 89 days
+
+With certbot, you should be able to run `certbot renew` when the time to renew comes up. There is also an [older
+version](https://github.com/singularityhub/sregistry/blob/master/scripts/generate_cert_tiny-acme.sh) that uses tiny-acme instead of certbot. For this second option, it basically comes down to:
 
  - starting nginx
  - installing tiny acme
@@ -96,53 +130,22 @@ Getting https certificates is really annoying, and getting `dhparams.pem` takes
  - moving them to where they need to be.
  - add a reminder or some other method to renew within 89 days
 
-Once you have done this, you should use the `docker-compose.yml` and the `nginx.conf` provided in the folder [https](https). So do something like this:
+Once you have done this (and you are ready for https), you should use the `docker-compose.yml` and the `nginx.conf` provided in the folder [https](https://github.com/singularityhub/sregistry/blob/master/https/). So do something like this:
 
 ```bash
 mkdir http
 mv nginx.conf http
 mv docker-compose.yml http
 
-mv https/docker-compose.yml $PWD
-mv https/nginx.conf $PWD
+cp https/docker-compose.yml .
+cp https/nginx.conf.https nginx.conf
 ```
 
-Most importantly, we use a text file to make sure that we generate a single certificate that covers both www* and without. This part of the [generate_cert.sh](https://github.com/singularityhub/sregistry/blob/master/scripts/generate_cert.sh) you will need to update the location (town, city, etc) along with your email and the domain you are using:
-
-```bash
-cat > csr_details.txt <<-EOF
-[req]
-default_bits = 2048
-prompt = no
-default_md = sha256
-req_extensions = req_ext
-distinguished_name = dn
- 
-[ dn ]
-C=US
-ST=California
-L=San Mateo County
-O=End Point
-OU=SingularityRegistry
-emailAddress=youremail@university.edu
-CN = www.domain.edu
- 
-[ req_ext ]
-subjectAltName = @alt_names
- 
-[ alt_names ]
-DNS.1 = domain.edu
-DNS.2 = www.domain.edu
-EOF
-```
-
-Specifically, pay close attention to the fields in the last two sections that need to be customized for the domain and region.
-
 If you run into strange errors regarding any kind of authentication / server / nginx when you start the images, likely it has to do with not having moved these files, or a setting about https in the [settings](https://github.com/singularityhub/sregistry/tree/master/shub/settings). If you have trouble, please post an issue on the [issues board](https://www.github.com/singularityhub/sregistry/issues) and I'd be glad to help.
 
 
 ## Build the Image (Optional)
-If you want to try it, you can build the image. Note that this step isn't necessary as the image is provided on [Docker Hub](https://hub.docker.com/r/vanessa/sregistry/). This step is optional - if you want to try building locally, you would do:
+If you want to try it, you can build the image. Note that this step isn't necessary as the image is provided on [Docker Hub](https://hub.docker.com/r/vanessa/sregistry/). This step is optional. However, if you are developing you likely want to build the image locally. You can do:
 
 
 ```bash