-
-
Notifications
You must be signed in to change notification settings - Fork 165
Docker newbie instructions
-- by Mike Gage
The docker files for webwork were constructed by Pan Luo (@xcompass) at UBC. Thanks Pan!!
This explanation is by and for newbies and may well be simplified, clarified, corrected in later editions. (Jan 6, 2018, July 21, 2018)
The purpose of using docker is to allow nearly everyone who has worked with webwork using the command line to download and test new patches or pull requests to WeBWorK. The initial setup takes a bit more than an hour. After that you can pull a new experimental version of WeBWorK in five minutes for review without interfering with any of the existing work on the computer. Faster testing will allow us to proceed more quickly with new WeBWorK features.
The ways of setting up Docker are evolving. What follows is a relatively simple (if slightly time consuming way) of installing WeBWorK inside docker. We are experimenting with installing webwork2, pg, the mysql database and OPL separately as docker components and then connecting them. This will make it easier to modify and save files in each of these components and to rebuild without having to rebuild everything. (Rebuilding the library database in particular takes some time.) Those with some experience using Docker and webwork may want to look at https://github.com/openwebwork/webwork2/wiki/Docker---things-to-get-used-to-(and-avoid-surprises) and review and contribute to experiments described in the develop branch and on the forum
I have run docker on the MacOS high sierra and am now running it on Mojave. You need at least El Capitan MacOS in order to run Docker.
Docker CE has been successfully installed on Linux machines.
Docker CE is available for Windows machines as well, but getting WeBWorK running under Docker CE for Windows is a bit more complicated than doing so on MacOS X or Linux.
- Docker CE for Windows depends on Hyper-V, so only works on Windows Professional, Education, or Enterprise, but not on Windows Home.
- There is an older Docker Toolbox for Windows which works differently, and which may work on on machines whose version of Windows does not support Docker CE for Windows.
- When Hyper-V is activated during the installation of Docker, it disables VirtualBox and other VM hypervisiors. (The cannot operate in parallel.)
- In order to run VirtualBox again you have to disable Hyper-V. To do so: uncheck Hyper-V in Control Panel -> Programs -> Programs and Features -> Turn Windows features on or off. Then reboot.
- Getting WeBWorK to work in Docker for Windows also requires:
- getting symbolic links to work properly in Windows (possible on recent versions of Windows when Developer Mode is enabled)
- and sharing the Windows drive on which you have your
webwork2folder.
- See:
Please report on your experiences. This wiki is versioned so don't worry about making mistakes when editing this document -- we can always restore an older version.
All commands are issued from the command line so you will need a terminal window. You will need some experience typing unix commands in order to follow these instructions.
Install Docker on your computer. (There are versions for Mac, Windows and Linux)
Docker download home Get the community edition. (CE)
The setup of docker on the Mac is standard and quick, see the Mac Instructions. For Windows see the Windows Instructions and https://servingmath.org/wp/2018/08/installing-webwork-on-docker-on-windows/
Test that it is installed with
$ docker --version
Docker version 18.06.0-ce, build 0ffa825
$ docker-compose --version
docker-compose version 1.22.0, build f46880f
$ docker-machine --version
docker-machine version 0.15.0, build b48dc28d
mkdir webwork_docker # (create a directory anywhere on your disk)
cd webwork_docker # all your work will be in this directory
docker run -d -p 80:80 --name webserver nginx # start a local webserver
after a few minutes an image of the nginx image will have been pulled from the cloud and a container running the webserver will have been created from the image.
In a web browser type http://localhost/ and return (see the Mac Instructions, Windows Instructions, pages above for expected response). Something along the lines of "welcome to nginx".
docker ps # see what containers are running, the first column is container ID
gage-imac-461:webwork_docker mgage$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
017118b37fef nginx "nginx -g 'daemon of�" 2 minutes ago Up 2 minutes 0.0.0.0:80->80/tcp webserver
docker logs CONTAINER_ID # print log/stdout of the container with CONTAINER_ID
gage-imac-461:webwork_docker mgage$ docker stop webserver
webserver
docker stop webserver # stop the server
docker ps # no container is running (they are all stopped and therefore "hidden").
docker ps -a # includes all of the containers including ones that are stopped.
gage-imac-461:webwork_docker mgage$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
017118b37fef nginx "nginx -g 'daemon of�" 5 minutes ago Exited (0) About a minute ago
webserver
docker start webserver # restart the server
docker rm -f webserver # remove the webserver (the -f stands for "force" )
# the image which you downloaded from the cloud and from which the container was created is still on your machine
docker images
gage-imac-461:webwork_docker mgage$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
nginx latest 7042885a156a 44 hours ago 109MB
docker rmi nginx # will remove the image completely
At this point you can read/skim the rest of the Mac Instructions, Windows Instructions, etc pages at your leisure. There are no changes that need to be made, but skimming to see what is available on this page can be helpful.
There is a second getting started page that is useful to read in order to get oriented. For example it explains the difference between a container and an image but first let's actually set up a webwork installation using docker.
- The two Git config settings included below
-c core.autocrlf=false -c core.symlinks=trueshould really only be needed on Windows (as these are the default values on Linux and MacOS X), however forcing them should not interfere with anything on Linux and MacOS X. - It is not necessary to install the
pgrepository below. You can skip thatgit cloneinstruction unless you plan to make changes to the corepgcode, in which case you will also need to editwebwork2/docker-compose.ymlto include your localpgtree at run-time. ( See https://github.com/openwebwork/webwork2/wiki/Docker---things-to-get-used-to-(and-avoid-surprises))
cd webwork_docker
ls # it should be empty -- running the demo above left no trace in this directory
git clone https://github.com/openwebwork/webwork2 -c core.autocrlf=false -c core.symlinks=true
# grab the webwork2 repo
#optional -- not needed until you start modifying pg
# git clone https://github.com/openwebwork/pg -c core.autocrlf=false -c core.symlinks=true
# grab and install the pg repo
cd webwork2
# Change to the branch needed. (Docker is now part of the master branch so no change is needed.
# If you are doing advanced work you may want to switch to the develop branch.
git branch
* master
# should show "* master" highlighted.
- Depending on how fast and how accurately you type these git commands will take 5 minutes or less.
- The revised files will run
OPL-updateautomatically when the container is first run.- this takes 20 minutes to complete.
- The revised files will run
Before proceeding, shut everything down in Docker and clear images. docker ps -a should be clear.
We are still in the webwork2 directory:
ls
If everything is successful the ls listing will show the usual webwork files but also the files
Dockerfile, docker-compose.yml and docker-entrypoint.sh
Now while still in the webwork2 directory type
docker build .
docker-compose up
You will now have to wait for a while. The terminal window will scroll through lots of lines to let you know that something is running.
Next time you will probably prefer to run
docker-compose up -d # detaches the terminal so you don't see all the output
# and so that you can quit the terminal without stopping the container.
You can see that it is installing a lot of things for the ubuntu version of the linux operating system.
Eventually it installs TeX and Apache, various CPAN modules (you'll see a lot of
Building and testing lines) and then MathJax. Then there are lines about creating the
mysql (dabase) application db_1 and finally the webwork application app_1. Then the admin
course is created and finally the lines will pause with "WebworkWebservice: webwork_directory is set to ..."
IMPORTANT: Don't close the terminal window since that will stop the container running the website.
Towards the end you should see
........
app_1 | Admin course is created.
app_1 | create modelCourse subdirectory
app_1 | defaultClasslist.lst is being created
app_1 | Library version is 2.5; using OPLtables!
app_1 | Mysql database reinitialized.
app_1 | Reading in textbook data from Textbooks in the library /opt/webwork/libraries/webwork-open-problem-library/OpenProblemLibrary.
app_1 | Reading in OPL taxonomy from Taxonomy2 in the library /opt/webwork/libraries/webwork-open-problem-library/OpenProblemLibrary.
app_1 | Saved taxonomy to /opt/webwork/webwork2/htdocs/DATA/tagging-taxonomy.json.
app_1 | Converting data from tagged pgfiles into mysql.
app_1 | Number of files processed:
app_1 | 100 200 300 400 500 600 700 800 900 1000
.....
as completes the installation of webwork (app_1), the database (db_1) and R (r_1) and it starts to build the OPL database. This takes the most time. There are about 35K files that have to be processed. Go get some coffee. After the files are processed there are 126 textbooks that need to be processed to line up problems relevant to those text books.
On my iMac the process took minutes from typing docker-compose up to the final result
app_1 | Updating Library Statistics.
app_1 | Importing statistics for 2 courses.
app_1 | 1 2
app_1 |
app_1 | mysql: [Warning] Using a password on the command line interface can be insecure.
app_1 |
app_1 | Done.
app_1 | Fixing ownership and permissions (just in case it is needed)
app_1 | AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.18.0.4. Set the 'ServerName' directive globally to suppress this message
app_1 | webwork.apache2-config: WeBWorK server is starting
app_1 | webwork.apache2-config: WeBWorK root directory set to /opt/webwork/webwork2 in webwork2/conf/webwork.apache2-config
app_1 | webwork.apache2-config: The following locations and urls are set in webwork2/conf/site.conf
app_1 | webwork.apache2-config: PG root directory set to /opt/webwork/pg
app_1 | webwork.apache2-config: WeBWorK server userID is www-data
app_1 | webwork.apache2-config: WeBWorK server groupID is root
app_1 | webwork.apache2-config: The webwork url on this site is http://localhost/webwork2
app_1 | webwork.apache2-config: The webwork smtp server address is localhost
app_1 | webwork.apache2-config: The webwork smtp server port is
app_1 | webwork.apache2-config: The webwork smtp server protocol is 'not ssl'
app_1 | WebworkSOAP::WSDL: webwork_directory set to /opt/webwork/webwork2 via $WeBWorK::Constants::WEBWORK_DIRECTORY set in webwork.apache2-config
app_1 | WebworkSOAP::WSDL: rpc_url set to http://localhost/webwork2_rpc
app_1 | WebworkWebservice: webwork_directory set to /opt/webwork/webwork2 via $WeBWorK::Constants::WEBWORK_DIRECTORY set in webwork.apache2-config
DO NOT CLOSE THIS WINDOW!!!! The window must be kept open while running the webserver. If you close the window it will shut down the webwork server.
Once the lines have stopped scrolling you can type
http://localhost:8080/webwork2 into your browser and you have a webwork site running.
The only course is "course administration" and it has login "admin" with password "admin".
You can login and add yourself as another admin in the admin course. (First add yourself as a student, then promote yourself to admin.)
You can check the library browser to make sure that it works and that the OPL database has been updated properly.
Create a new course 'test_course', add some guest "students" by importing from demoCourse.lst and add a few homework problems to the course using the LibraryBrowser.
I forgot to select a target set from the course in the library browser. Usually the blank for the target set would read "Select a Set from this Course" but because no homework sets had been set up at all this field was completely blank. This lead to an "instructorXMLRPC: 403 Forbidden" error. Usually this error is triggered by some glitch in setting up the site URL so that it works properly with AJAX, but not in this case. Making sure that I had created a new set in the course and then choosing that set as a target was enough to clear the error. I found a message in the error log at /var/log/apache2/error.log indicating that no homework set had been created for the course (see more information below on reading this log) but that message was not transmitted to the screen. It's likely that this particular error can only occur with a brand new, empty course.
In a new terminal window go to webwork_docker/webwork2. Don't close the first terminal window.
Where is the data being stored? You won't be able to see much of the data that you usually expect to see stored in webwork on the disk, at least not from "outside" the docker environment. One thing you can see is the courses directory.
A new directory has been created to hold persistent webwork2 data
cd webwork_docker
ls ww-webwork-data
courses
ls ww-docker-data/courses
admin/ adminClasslist.lst defaultClasslist.lst modelCourse/ test_course/
You'll see courses, which is where persistent data for courses run in the Docker container will be saved. You can add (upload) archived courses to the courses directory and then unarchive them using the admin course in the browser.
With older versions of the Docker config files, there will also be a db directory which is the location for the persistent storage for the database. More recent versions of the Docker config files put the databases into a named Docker volume. It's wise not to touch the db directory unless you are a MariaDB expert.
Read more about the persistent and transient storage at https://github.com/openwebwork/webwork2/wiki/Docker---things-to-get-used-to-(and-avoid-surprises). It's one of the things that takes a while to get used to. The exact balance of which data is persistent and which is transient will continue to be tweaked as we work on creating the best experience of using docker with WeBWorK.
In your second terminal window (in directory webwork_docker/webwork2) -- you may have to cd to that directory -- type
docker-compose down
Stopping webwork2_app_1 ... done
Stopping webwork2_db_1 ... done
Stopping webwork2_r_1 ... done
Removing webwork2_app_1 ... done
Removing webwork2_db_1 ... done
Removing webwork2_r_1 ... done
If you go back to your first window you will see that the build process has exited and your browser
can no longer access the webwork site.
## congratulations you've completed your first use of webwork on your laptop using docker.
## Next try -- things will be quicker now:
In the first window type
docker-compose up -d gage-imac-461:webwork2 mgage$ docker-compose up -d Creating network "webwork2_default" with the default driver Creating webwork2_db_1 ... done Creating webwork2_r_1 ... done Creating webwork2_app_1 ... done
Notice that things are much quicker this time as the images are already built. And the webserver has started up right where you left off.
Close the first terminal window. And check that the website is still running. (This is what the -d accomplished.)
In the second terminal (webwork_docker/webwork2 directory) type
docker container ls # (or docker ps) and observer that there are two containers powering the website: webwork and mariadb(Mysql)
Observe the container id for webwork. Some commands will require at least the first 4 or 5 characters of that WeBWorK_id to identify the container. Sometimes the name "webwork" alone will work.
type
docker container exec -it WeBWorK_id bash root@WeBWorK_id:/opt/webwork#
A terminal window has just been opened inside the "remote" machine at the directory `/opt/webwork`.
cd webwork2/bin ./OPL-update
and go get a cup of coffee while the database for the OPL tags are created.
You will get some reassuring lines about how many problems have been processed
but there are 35K of them so this process takes a while.
When it's finally done type
`exit` and you will close the terminal window "connection" to the container (the remote machine)
-----------------
*This takes about an hour from beginning the git downloads to this point. A good twenty minutes
of that was waiting for the OPL-update script to populate the database with the problem tags.*
-------------------
### Getting to the command line in the Docker container
In a second terminal (webwork_docker/webwork2 directory) type
docker container ls # (or docker ps) and observer that there are three containers powering the website: webwork and mariadb(Mysql) and r
Observe the container id for webwork. Some commands will require at least the first 4 or 5 characters of that WeBWorK_id to identify the container. Sometimes the name `webwork` alone will work.
type
docker container exec -it WeBWorK_id bash root@WeBWorK_id:/opt/webwork#
A terminal window has just been opened inside the Docker container (as if it were a "remote" machine) with the current directory set to `/opt/webwork`.
Anything that you would do by logging in to your local WeBWorK server on the command line
you can do by "logging" in to the container using the approach above.
**Warning:** Recall that changes made in the command line of the Docker container will only be persistent if they are making changes to files on the persistent storage. See <https://github.com/openwebwork/webwork2/wiki/Docker---things-to-get-used-to-(and-avoid-surprises)>
### Installing an archived course:
If you have a copy of an archived course that you've downloaded from your usual webwork site (a .gz file)
we can place that in the container as follows.
1. In webwork_docker/webwork2
docker-compose down
cp ../path/to/course/archive .data/courses
ls .data/courses # just to make sure the course is there
ls .data # should have courses and db
docker-compose up
2. Check the web browser and the admin course. You should be able to "unarchive" your course using
the "unarchive" tab.
3. If it is an old course you may have to "update courses" to fix the database
on the course you have imported.
4. In the web browser open the library browser and make sure that it works.
5. You can now add problems to your existing course or add
### modelCourse
* The recent Docker control files set up the modelCourse so if you used them you can **ignore this section**.
* The older Docker control files **do not**... so in that case:
* Let's do one more thing so that it is easier to create new courses in your local website.
* One way we can fix that is this:
docker container exec -it webwork2_app_1 bash # open a terminal into the container
ls # should see toplevel files of webwork cd courses ls # should see /admin course ln -s ../webwork2/courses.dist/modelCourse/ modelCourse # link to the modelCourse distributed in webwork2
### Misc:
Double check that you are running the versions of webwork and pg that you want. At the bash prompt
(which you opened inside the Docker container) type:
cd pg git branch -a # if you are not seeing rel-2.14 and you want to use it git branch -t rel-2.14 origin/rel-PG-2.14 git branch git checkout rel-2.14 apachectl graceful # restarts the web server.
### Adding a new course
* Now try adding a new course from the browser using the Course Administration web page.
* You should see that you can build it using modelCourse as a template.
### Shutting down (and restarting) the Docker container:
* When you are done using your Docker container type `docker-compose down` and docker will stop using CPU's on your computer.
* It will still take up some disk memory space.
* It can be restarted with `docker-compose up -d `.
### Final comments:
* see Docker for newbies part 2 for adding daemon_course
That's enough for now. Play with the WeBWorK site you have created and go back to skimming
the [getting started](https://docs.docker.com/get-started/) to try to figure out what you have just
done. We've only used the first two sections of that document -- there is more that you can do to
create a "production site" that can handle many students.
**Warning** With what we have so far you **DO NOT** want to use this a production WeBWorK site -- this is a local installation on your machine for you only. None of the databases are using passwords for example, so there is not much security if you start letting people do homework on a network connected to your computer.
To repeat, the purpose of using Docker (so far) is to allow nearly everyone who has worked with WeBWorK using the command line to download and test new patches or pull requests to WeBWorK. Faster testing will allow us to proceed more quickly with new WeBWorK features.
The UBC people are eventually planning to use docker on their production machines so that it is super simple to install WeBWorK.
<!--Also watch out, this installation temporary. If you create a WeBWorK course in your local site it will stay around if you do docker-compose down, which just makes the container dormant. It will also stay around if you quit the Docker app. If you remove the container however (a command with a "rm" in it) the site will disappear. You could first save the course using the "archive" button in the FileManager in WeBWorK and download the archived file. (You could also do this for multiple courses from Course Administration course.) -->
## Cleaning up
All the persistent files (database and course files) are stored in `webwork_docker/webwork2/.data` directory (or a named volume for database storage in more recent configurations). In order to do a completely clean uninstall of this "webwork in a box", remove the `.data` directory or `webwork_docker` directory (and the named `mysql` storage volume).