Merge pull request #31 from EarthCubeInGeo/mnt_update_windows_docs

[MNT]: Update install and usage docs

README.rst

@@ -2,7 +2,7 @@ Resen
 *****
 Resen (REproducible Software ENvironment), is a tool that enables reproducible scientific data analysis, built using python and docker. It is designed to make it easier for geospace researchers to share analysis and results, as well as build off of work others have done. Resen was developed under the InGeO project, currently supported by the National Science Foundation's Cyberinfrastructure for Sustained Scientific Innovation (CSSI) program (Grant \#1835573). For more information about the InGeO project, please visit the `InGeO website <https://ingeo.datatransport.org>`_.
 
-.. image:: resen_consept.png
+.. image:: images/resen_concept.png
 
 Resen is based on the concept of portable environments, or buckets, where code can be developed and run independent of a users system. When you start a resen bucket, it has a variety of common geospace software packages preinstalled and ready for use. This means you have easy access to common models and datasets, and can start using them in your analysis immediately. You can also set up your bucket to access your own datasets, locally stored on your machine.
 

docs/conf.py

@@ -18,11 +18,11 @@
 # -- Project information -----------------------------------------------------
 
 project = 'Resen'
-copyright = '2019, A. Reimer'
-author = 'A. Reimer'
+copyright = '2019, Integrated Geoscience Observatory'
+author = 'Resen Team'
 
 # The full version, including alpha/beta/rc tags
-release = 'v2019.1.0rc1'
+release = 'v2019.1.0rc2'
 
 master_doc = 'index'
 
@@ -37,6 +37,9 @@
  'sphinx.ext.napoleon',
 ]
 
+
+default_role = "autolink"
+
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
 

docs/index.rst

@@ -1,23 +1,11 @@
-.. Resen documentation master file, created by
- sphinx-quickstart on Mon May 13 11:49:19 2019.
- You can adapt this file completely to your liking, but it should at least
- contain the root `toctree` directive.
-
+########
 Contents
-********
+########
 
 .. toctree::
- :maxdepth: 2
+ :maxdepth: 3
 
  readme
- installation
+ installation/index
  usage
 
-
-
-Indices and tables
-==================
-
-* :ref:`genindex`
-* :ref:`modindex`
-* :ref:`search`

docs/installation/index.rst

@@ -0,0 +1,11 @@
+############
+Installation
+############
+
+.. toctree::
+ :maxdepth: 3
+
+ installation.general
+ installation.windows
+
+

docs/installation.rst → docs/installation/installation.general.rst

@@ -1,5 +1,5 @@
-Installation
-************
+Gerneral Instructions
+*********************
 
 Resen is built off of both python 3 and docker, so you must have both of these installed for Resen to function.
 

docs/installation/installation.rst

@@ -0,0 +1,8 @@
+############
+Installation
+############
+
+.. toctree::
+ :maxdepth: 2
+ installation.general
+ installation.windows

docs/installation/installation.windows.rst

@@ -0,0 +1,77 @@
+Windows Gotchas
+***************
+
+Resen requires both python 3 and docker to function. Here we provide a basic guide for installing both python 3 and docker. We have tested this procedure and know it works. Python 3 and Resen are easy to install. Docker is also fairly easy, but there are some subtle details that need to be emphasized for a smooth installation process.
+
+Install Anaconda and Resen
+==========================
+
+**Anaconda**:
+We recommend downloading and installing the Python 3 Anaconda Distribution (https://www.anaconda.com/distribution/). This simplifies the installation and usage of several common software tools needed to install and run Resen.
+
+**Resen**:
+Using the start menu search, open the "Anaconda Powershell Prompt" and navigate to a directory where you wish to host the Resen source code. Next, install Resen by first cloning the resen GitHub repo (https://github.com/EarthCubeInGeo/resen)::
+
+ git clone https://github.com/EarthCubeInGeo/resen.git
+
+Change into the ``resen`` directory::
+
+ cd resen
+
+Finally, install Resen::
+
+ pip install .
+
+Once complete, this will provide the command line command ``resen``. Next, we need to install Docker.
+
+
+
+Docker
+======
+
+For Windows, there are 2 options for installing Docker, which depends on what else you use and do with your Windows system::
+
+1. `Docker Desktop for Windows <https://docs.docker.com/docker-for-windows/install/>`_
+
+2. `Docker Toolbox <https://docs.docker.com/toolbox/toolbox_install_windows/>`_
+
+If you use `Oracle VM VirtualBox <https://www.virtualbox.org/>`_ to run virtual machines on your Windows system, DO NOT install Docker Desktop. You must instead install Docker Toolbox. Docker Desktop uses Hyper-V, which is not compatible with VirtualBox.
+
+**Docker Desktop**
+TODO. If you can help fill this in, please make a PR to the develop branch for resen!
+
+**Docker Toolbox**
+Docker Toolbox essentially works by running docker inside of a Linux virtual machine using VirtualBox. The VM that gets installed is name "default" and we will refer to this "default" Docker virtual machine as the "Docker VM". To install it Docker Toolbox, do the following:
+
+1. Shutdown any VirtualBox VMs that you currently have running and take note of the VirtualBox version you have installed. Docker Desktop installs an older version of VirtualBox on your system, but this version you are currently running can be upgraded back to the version you are currently running.
+
+2. `Follow the instructions on here to install Docker Toolbox <https://docs.docker.com/toolbox/toolbox_install_windows/>`_. Once installed, restart your computer and then run the ``Docker Quickstart Terminal`` from the start menu. ``TODO: insert screenshot``
+
+3. Now we need to add port forwarding and check the shared folders for the Docker VM in VirtualBox. To do this, open VirtualBox and open the "Settings" for the "default" VM, like so:
+
+.. image:: images/vbox.png
+
+Add a new port forwarding rule by navigating to Settings->Network->Adapter 1->Advanced->Port Forwarding:
+
+.. image:: images/port_forward.png
+
+Here, we need to add a port forwarding rule for each bucket we create in Resen. Resen requires port 9000 for one bucket and then increments by 1 for every new bucket created. This means that if you have 5 buckets, you will need to make a port forward rule for ports 9000, 9001, 9002, 9003, and 9004. Change both the Host and Guest Ports as seen in the above screenshot.
+
+Now we can optionally add Shared Folders. By default, Docker Toolbox shares the ``C:\Users`` directory with the Docker VM at ``/c/Users``. This means that directories in ``C:\Users`` will be available to mount into a Resen bucket via the ``/c/Users`` Shared Folder in VirtualBox. If additional shared directory locations are desired add them. For example:
+
+.. image:: images/shared_folder.png
+.. image:: images/add_shared_folder.png
+
+makes an additional location, ``D:\ashto`` available to the Docker VM at the location ``/d/ashto`` so that any directories in ``D:\ashto`` can be mounted into a resen bucket via ``/d/ashto``. After adding or removing Shared Folders, you must restart the Docker VM. This can be done by running:
+
+ docker-machine restart
+
+in the "Docker Quickstart Terminal".
+
+4. Optionally, you can now re-install the newer verions of VirtualBox that you had previously installed. Before doing this, shutdown the Docker Toolbox VM. After re-installing VirtualBox, restart your computer and then open the "Docker Quickstart Terminal" again.
+
+**Running Resen**
+
+Now you can run Resen! To do this, open an "Anaconda Powershell Prompt" and type "resen" and hit enter! You should see something similar to:
+
+.. image:: images/resen_cmd.png

docs/usage.rst

@@ -12,7 +12,7 @@ This will open the resen tool::
  | / _|\__ \ _|| .` |
  |_|_\___|___/___|_|\_|
  
- Resen 2019.1.0rc1 -- Reproducible Software Environment
+ Resen 2019.1.0rc2 -- Reproducible Software Environment
  
  [resen] >>> 
 
@@ -22,97 +22,117 @@ Type ``help`` to see available commands::
 
 This will produce a list of resen commands you will use to manage your resen buckets::
 
- Documented commands (type help <topic>):
- ========================================
- EOF add_storage help remove_port start_jupyter
- add_image create_bucket quit remove_storage status 
- add_port exit remove_bucket start_bucket stop_bucket 
+ Documented commands (type help <topic>):
+ ========================================
+ EOF exit quit start_jupyter stop_jupyter
+ create_bucket help remove_bucket status
 
 To get more information about a specific command, enter ``help <command>``.
 
 Resen Workflow
 ==============
 
-Use Resen to start, stop, and manage resen buckets.  Buckets are portable, system independent environments where code can be developed and run.  Buckets can be shared between systems and the code in it will run exactly the same way.  Resen buckets come preinstalled with a variety of common geospace software that can be used immediately in analysis.
+Use Resen to create and remove buckets. Buckets are portable, system independent environments where code can be developed and run. Buckets can be shared between Windows, Linux, and macos systems and all analysis within the bucket will be run exactly the same. Resen buckets come preinstalled with a variety of common geospace software that can be used immediately in analysis.
 
 Setup a New Bucket
 ------------------
 
-1. Create a new bucket. Buckets should be given a name that a string of less than 20 characters with no spaces. To create a bucket named ``amber``::
+1. Creating a new bucket is performed with the command::
 
- [resen] >>> create_bucket amber
+ [resen] >>> create_bucket
 
-2. Add a resen image to the bucket. The image name must be a valid resen-core tag that is available on Docker Hub. For a list of tags, see https://hub.docker.com/r/earthcubeingeo/resen-core/tags. To add the ``2019.1.0rc1`` image to ``amber``::
+ The ``create_bucket`` command queries the user for several pieces of information required to create a bucket. First it asks for the bucket name. Creating a bucket named ``amber``::
 
- [resen] >>> add_image amber 2019.1.0rc1
+ Please enter a name for your bucket.
+ Valid names may not contain spaces and must start with a letter and be less than 20 characters long.``
+ >>> Enter bucket name: amber
 
-3. Add a port for the bucket. This is required for the bucket to run jupyter notebooks. A port on the bucket will map to a port on the local machine. Map the local port 8000 to the bucket port 8080::
+ Next, the user is asked to specify the version of resen-core to use::
 
- [resen] >>> add_port amber 8000 8080
+ Please choose a version of resen-core.
+ Available versions: 2019.1.0rc2
+ >>> Select a version: 2019.1.0.rc2
 
-4. Add a persistent storage directory to the bucket. This lets the bucket access code and data on the local machine. Files created in a Resen bucket will be available outside of the bucket or after the bucket has been deleted ONLY if they are saved in a persistent storage directory. Multiple storage directories can be added to the bucket. Each persistent storage directory can have either read (`r`) or read/write (`rw`) permissions, specifying whether or not resen can write to the local directory. Add the local directories ``/home/usr/code/fossil`` and ``/home/usr/data`` to the bucket in the ``/home/jovyan/work`` directory::
+ Optionally, one may then specify a local directory to mount into the bucket at ``/home/jovyan/work``::
 
- [resen] >>> add_storage amber /home/usr/code/fossil /home/jovyan/work/fossil rw
- [resen] >>> add_storage amber /home/usr/data /home/jovyan/work/data r
+ Local directories can be mounted to either /home/jovyan/work or /home/jovyan/mount/ in
+ a bucket. The /home/jovyan/work location is a workspace and /home/jovyan/mount/ is intended
+ for mounting in data. You will have rw privileges to everything mounted in work, but can
+ specify permissions as either r or rw for directories in mount. Code and data created in a
+ bucket can ONLY be accessed outside the bucket or after the bucket has been deleted if it is
+ saved in a mounted local directory.
+ >>> Mount storage to /home/jovyan/work? (y/n): y
+ >>> Enter local path: /some/local/path
 
-5. Check the status of the bucket::
+ Followed by additional local directories that can be mounted under ``/home/jovyan/mount``::
 
- [resen] >>> status amber
- {'bucket': {'name': 'amber'}, 'docker': {'image': 'docker.io/earthcubeingeo/resen-core:2019.1.0rc1', 'container': None, 'port': [[8000, 8080, True]], 'storage': [['/home/usr/code/fossil', '/home/jovyan/work/fossil', 'rw'], ['/home/usr/data', '/home/jovyan/work/data', 'ro']], 'status': None}}
+ >>> Mount storage to /home/jovyan/mount? (y/n): y
+ >>> Enter local path: /some/other/local/path
+ >>> Enter bucket path: /home/jovyan/mount/data001
+ >>> Enter permissions (r/rw): r
+ >>> Mount additional storage to /home/jovyan/mount? (y/n): n
 
-At this point, the bucket should have a name, an image, at least one port, and at least one storage location. Status should be ``None``.
+ Finally, the user is asked if they want jupyterlab to be started::
 
-Work in a Bucket
-----------------
-1. Check what buckets are available with ``status``::
+ >>> Start bucket and jupyterlab? (y/n): y
+
+ after which resen will begin creating the bucket. Example output for a new bucket named ``amber`` with jupyterlab started is::
+
+ ...adding core...
+ ...adding mounts...
+ Bucket created successfully!
+ ...starting jupyterlab...
+ Jupyter lab can be accessed in a browser at: http://localhost:9000/?token=61469c2ccef5dd27dbf9a8ba7c296f40e04278a89e6cf76a
+
+2. Check the status of the bucket::
 
- [resen] >>> status
- Bucket Name Docker Image Status 
- amber docker.io/earthcubei... None 
+ [resen] >>> status amber
+ {'bucket': {'name': 'amber'}, 'docker': {'image': '2019.1.0rc2', 'container': 'a6501d441a9f025dc7dd913bf6d531b6b452d0a3bd6d5bad0eedca791e1d92ca', 'port': [[9000, 9000, True]], 'storage': [['/some/local/path', '/home/jovyan/work', 'rw'], ['/some/other/local/path', '/home/jovyan/mount/data001', 'ro']], 'status': 'running', 'jupyter': {'token': '61469c2ccef5dd27dbf9a8ba7c296f40e04278a89e6cf76a', 'port': 9000}, 'image_id': 'sha256:3ba43e401c1b1a8eca8969aec8426a22d99bca349fd837270fa06dbcaefaeb47', 'pull_image': 'earthcubeingeo/resen-core@sha256:c3783e3b7f05ec17f9381a01009b794666107780d964e8087c62f7baaa00049d'}}
 
-Newly created buckets that have not been started will have Status ``None``.
+At this point, the bucket should have a name, an image, at least one port, and optionally one or more storage location. Status should be ``running`` if the user decided to have jupyterlab started, otherwise the status will be ``None``.
+
+Work with a Bucket
+------------------
+1. Check what buckets are available with ``status``::
 
-2. Start a newly created bucket or restart a bucket that has been exited::
+ [resen] >>> status
+ Bucket Name Docker Image Status
+ amber 2019.1.0rc2 running
 
- [resen] >>> start_bucket amber
- Pulling image: docker.io/earthcubeingeo/resen-core:2019.1.0rc1
- This may take some time...
- Done!
+ If a bucket is running, it will consume system resources accordingly.
 
-The status of ``amber`` should now be ``running``::
+2. Stop jupyter lab from a bucket::
 
- [resen] >>> status
- Bucket Name Docker Image Status 
- amber docker.io/earthcubei... running 
+ [resen] >>> stop_jupyter amber
 
-3. Use the bucket to start a jupyter server. Make sure to include the local port and the bucket port that forwards to it. Start a jupyter server in ``amber``::
+ The status of ``amber`` should now be ``exited``::
 
- [resen] >>> start_jupyter amber 8000 8080
+ [resen] >>> status
+ Bucket Name Docker Image Status
+ amber 2019.1.0rc2 exited
 
-The jupyter server starts in the ``/home/jovyan/work`` directory, which should include the persistent storage directories ``fossil`` and ``data``. Alternatively you can start directly a jupyter lab adding ``--lab`` to the previous command::
+ The bucket will still exist and can be restarted at any time, even after quitting and restarting resen.
 
- [resen] >>> start_jupyter amber 8000 8080 --lab
- 
-or, if you already started the notebook without ``--lab`` you can change the url in your browser from ``http://localhost:8000/tree`` to ``http://localhost:8000/lab``. One can go back from the lab to the notebook through Menu -> Help -> Launch Classic Notebook.
+3. Start a jupyter lab in bucket ``amber`` that has been stopped::
 
-4. Stop the jupyter server by clicking the "Quit" button on the home page of the notebook. The jupyter lab "Quit" button has not been configured in this version.
+ [resen] >>> start_jupyter amber
 
-5. Stop the bucket::
+ The status of ``amber`` should now be ``running``::
 
- [resen] >>> stop_bucket amber
+ [resen] >>> status
+ Bucket Name Docker Image Status
+ amber 2019.1.0rc2 running
 
-The status of ``amber`` should now be ``exited``::
 
- [resen] >>> status
- Bucket Name Docker Image Status 
- amber docker.io/earthcubei... exited 
+ The jupyter lab server starts in the ``/home/jovyan`` directory, which should include the persistent storage directories ``work`` and ``mount``.
+ The user can alternate between the jupyter lab and the classic notebook view by changing the url in the browser from ``http://localhost:8000/lab`` to ``http://localhost:8000/tree``. Alternatively one can switch from the lab to the notebook through Menu -> Help -> Launch Classic Notebook.
 
-The bucket will still exist and can be restarted at any time, even after quitting and restarting resen.
 
 Remove a Bucket
 ---------------
-Delete a bucket::
+The user can delete a bucket with the following command::
 
- remove_bucket amber
+ [resen] >>> remove_bucket amber
 
-WARNING: This will permanently delete the bucket. Any work that is not saved in a persistent storage directory will be lost.
+A bucket that is running needs to be stopped before removed.
+WARNING: This will permanently delete the bucket. Any work that was not saved in a mounted storage directory will be lost.