Merge pull request #62 from cedadev/development

Pull development into main ahead of tagged release

docs/source/conf.py

@@ -54,4 +54,10 @@
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = ['_static']
+html_static_path = ['_static']
+
+html_logo = "ceda.png"
+html_theme_options = {
+    'logo_only': True,
+    'display_version': False,
+}

docs/source/cta-emulator.rst

@@ -0,0 +1,187 @@
+CERN Tape Archive Set Up
+========================
+
+As part of the development of the NLDS, a tape emulator was set up to better 
+understand how to interact with ``xrootd`` and the CERN tape archive. The 
+following instructions detail how to set up such a tape emulator and are adapted 
+from the instructions on [the CTA repo]
+(https://gitlab.cern.ch/cta/CTA/-/tree/main/continuousintegration/buildtree_runner) 
+and those provided by STFC's Scientific Computing Department at the Rutherford 
+Appleton Laboratory. There are two major parts: commissioning the virtual 
+machine, and setting it up appropriately according to the CERN instructions. 
+
+
+Commissioning the VM
+--------------------
+
+These instructions are specifically for commissioning a VM on the STFC openstack
+cloud interface. For other machines, slightly different instructions will need 
+to be followed. 
+
+After you have logged in to the openstack interface, you click on the "Launch 
+Instance" button on the top to create the VM and then:
+
+1. In the "Details" tab, give your VM a suitable name
+2. In the "Source" tab, select scientificlinux-7-aq as a source image
+3. In the "Flavour" tab, select a VM type depending on how many VCPUs, RAM and 
+   disk size you need
+4. In the "Networks" tab, select "Internal"
+5. In the "Key Pair" tab, upload your public rsa ssh key so you can login to the 
+   VM once it is created.
+6. In the "Metadata" tab, click on the pull-down menu "Aquilon Image Properties" 
+   and then set it to "Aquilon Archetype", specifying ``ral-tier1``, and also 
+   "Aquilon Personality" specifying it as ``eoscta_ci_cd``. Note that you will 
+   have to manually write these values out so it's worth copy pasting to avoid 
+   typos!
+7. Press the "Launch Instance" button and the VM will be created. Give it some 
+   time so that quattor runs - quattor being a vm management tool like Puppet. 
+   It may also need a reboot at some point.
+
+This setup produces a vm which requires logging in as your openstack username - 
+in most cases this will be your STFC federal ID. You will be able to sudo 
+assuming the machine remains in the correct configuration.
+
+**Note:**
+The above setup is one that theoretically works. However, the machine which – 
+after some attempts – successfully had CTA installed on it had to be 
+commissioned manually by SCD so that  
+
+(a) quattor could be made sure to have run successfully and then subsequently 
+    disabled 
+(b) I would be able to log in as ``root`` thus negating the need to edit the 
+    sudoers file after each reboot. 
+
+I would strongly recommend this approach if SCD are agreeable to commissioning 
+your vm for you. 
+
+
+Setting up CTA on the VM
+------------------------
+
+The following are the working set of instructions at time of writing, provided 
+by SCD at RAL. 
+
+* **Ensure quattor is not running**
+
+  There should be an empty file at ``/etc/noquattor``, if there is not one then 
+  create it with 
+
+  ``sudo touch /etc/noquattor``
+
+* **Clone the CTA gitlab repo**
+
+  ``git clone https://gitlab.cern.ch/cta/CTA.git``
+
+* **User environment**
+
+  As per instructions
+
+  ``cd ./CTA/continuousintegration/buildtree_runner/vmBootstrap``
+
+  BUT in bootstrapSystem.sh, delete/comment line 46 and then
+
+  ``./bootstrapSystem.sh``
+
+  When prompted for password, press return (i.e don't give one). This creates 
+  the ``cta`` user and adds them to sudoers
+
+
+* **CTA build tree**
+
+  As per instructions
+
+  ``su - cta``
+  ``cd ~/CTA/continuousintegration/buildtree_runner/vmBootstrap``
+
+  BUT edit lines 54,55 in bootstrapCTA.sh to look like
+
+  ``sudo wget https://public-yum.oracle.com/RPM-GPG-KEY-oracle-ol7 -O /etc/pki/rpm-gpg/RPM-GPG-KEY-oracle --no-check-certificate``
+  ``sudo wget https://download.ceph.com/keys/release.asc -O /etc/pki/rpm-gpg/RPM-ASC-KEY-ceph --no-check-certificate``
+
+  Note the change in the URL on line 55 from ``git.ceph.com`` to 
+  ``download.ceph.com``, as well as the addition of the ``--no-check-certificate`` 
+  flag. 
+
+  Then run bootstrapCTA.sh (without any args)
+
+  ``./bootstrapCTA.sh``
+
+
+* **Install MHVTL**
+
+  As per instructions
+
+  ``cd ~/CTA/continuousintegration/buildtree_runner/vmBootstrap``
+  ``./bootstrapMHVTL.sh``
+
+* **Kubernetes setup**
+
+  As per instructions
+
+  ``cd ~/CTA/continuousintegration/buildtree_runner/vmBootstrap``
+  ``./bootstrapKubernetes.sh``
+
+  and reboot host
+
+  ``sudo reboot``
+
+* **Docker image**
+
+  Depending on how your machine was set up you may now need to ensure that 
+  quattor is still disabled (i.e. that the ``/etc/noquattor`` file still exists) 
+  and that the cta user is still in the sudoers file. This will not be necessary 
+  if you are running as ``root``.
+
+  Then, as per instructions
+
+  ``su - cta``
+  ``cd ~/CTA/continuousintegration/buildtree_runner``
+
+  BUT edit lines 38,39 in  /home/cta/CTA/continuousintegration/docker/ctafrontend/cc7/buildtree-stage1-rpms-public/Dockerfile to look like
+
+  ``RUN wget https://public-yum.oracle.com/RPM-GPG-KEY-oracle-ol7 -O /etc/pki/rpm-gpg/RPM-GPG-KEY-oracle --no-check-certificate``
+  ``RUN wget https://download.ceph.com/keys/release.asc -O /etc/pki/rpm-gpg/RPM-ASC-KEY-ceph --no-check-certificate``
+
+  then run the master script to prepare all the Docker images. 
+
+  ``./prepareImage.sh``
+
+* **Preparing the environment (MHVTL, kubernetes volumes...)**
+
+  As per instructions
+
+  ``cd ~/CTA/continuousintegration/buildtree_runner``
+  ``sudo ./recreate_buildtree_running_environment.sh``
+
+* **Preparing the CTA instance**
+
+  As per instructions
+
+  ``cd ~/CTA/continuousintegration/orchestration``
+  ``sudo ./create_instance.sh  -n cta -b ~ -B CTA-build -O -D -d internal_postgres.yaml``
+
+  This may work first time but it never did for me, so the fix is to then run 
+
+  ``./delete_instance.sh -n cta``
+
+  To remove the instance and then re-create it with the same command as above
+  ``sudo ./create_instance.sh  -n cta -b ~ -B CTA-build -O -D -d internal_postgres.yaml``
+
+  This can be verified to be working with a call to 
+
+  ``kubectl -n cta get pods``
+
+  which should return a list of the working pods, looking something like:
+
+    ============  ========  ========  =========  ===
+    NAME          READY     STATUS    RESTARTS   AGE
+    ============  ========  ========  =========  ===
+    client        1/1       Running   0          35m
+    ctacli        1/1       Running   0          35m
+    ctaeos        1/1       Running   0          35m
+    ctafrontend   1/1       Running   0          35m
+    kdc           1/1       Running   0          35m
+    postgres      1/1       Running   0          36m
+    tpsrv01       2/2       Running   0          35m
+    tpsrv02       2/2       Running   0          35m
+    ============  ========  ========  =========  ===

docs/source/index.rst

@@ -8,14 +8,22 @@ Welcome to Near-line Data Store's documentation!
 
 .. toctree::
    :maxdepth: 2
-   :caption: Contents:
+   :caption: Contents
 
-   Getting started<home>
+   Getting started <home>
    Specification <specification>
    NLDS Server API <nlds-server>
    NLDS Processors API <nlds-processors>
 
 
+.. toctree::
+   :maxdepth: 2
+   :caption: Advanced
+
+   The server config file <server-config/server-config>
+   Server config examples <server-config/examples>
+   Setting up a CTA tape emulator <cta-emulator>
+
 
 Indices and tables
 ==================

docs/source/nlds-processors.rst

@@ -1,5 +1,7 @@
-Core content of the nlds-processors
-===================================
+Microservices
+=============
+
+Core content of the nlds-processors.
 
 The Consumer class
 ------------------
@@ -10,21 +12,42 @@ The Consumer class
 
 The processors
 --------------
-Also referred to as 'microservices'
+Also referred to as 'microservices', 'consumers', or 'workers':
 
 .. automodule:: nlds_processors.nlds_worker
     :members:
 
 .. automodule:: nlds_processors.index
     :members:
 
-.. automodule:: nlds_processors.transfer
+.. automodule:: nlds_processors.transferers.base_transfer
+    :members:
+
+.. automodule:: nlds_processors.transferers.put_transfer
+    :members:
+
+.. automodule:: nlds_processors.transferers.get_transfer
+    :members:
+
+.. automodule:: nlds_processors.db_mixin
+    :members:
+
+.. automodule:: nlds_processors.catalog.catalog_models
+    :members:
+
+.. automodule:: nlds_processors.catalog.catalog
+    :members:
+
+.. automodule:: nlds_processors.catalog.catalog_worker
+    :members:
+
+.. automodule:: nlds_processors.monitor.monitor_models
     :members:
 
-.. automodule:: nlds_processors.catalog
+.. automodule:: nlds_processors.monitor.monitor
     :members:
 
-.. automodule:: nlds_processors.monitor
+.. automodule:: nlds_processors.monitor.monitor_worker
     :members:
 
 .. automodule:: nlds_processors.logger

docs/source/nlds-server.rst

@@ -1,5 +1,7 @@
-Core content of the nlds-server
-===============================
+NLDS API-server
+===============
+
+The core content of the NLDS API-server run using FastAPI. 
 
 The Publisher class
 -------------------
@@ -28,7 +30,7 @@ The authenticators
     :members:
     :undoc-members:
 
-Authenicate methods also contains 3 general methods, used by the above 2 
+Authenticate methods also contains 3 general methods, used by the above 2 
 modules, to validate the given user, group and token.