upgrade to irods v4.3.3 (#16, #18, #19, #21, #24, #27)

bihealth · Oct 22, 2024 · 8ecc563 · 8ecc563
1 parent 0a29225
commit 8ecc563
Show file tree

Hide file tree

Showing 9 changed files with 220 additions and 282 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -0,0 +1,16 @@
+# Changelog
+
+## Unreleased
+
+- Upgrade to iRODS v4.3.3 (#16)
+- Upgrade to PostgreSQL >11 (#18)
+- Upgrade image to Ubuntu 20.04 (#19)
+- Upgrade scripts for Python3 (#21)
+- Set up logging with syslog (#16)
+- Enable setting `irods-rule-engine-plugin-python` version in `build.sh` (#27)
+- Add changelog (#22)
+- Change custom SODAR PAM login method from `POST` to `GET` (bihealth/sodar-server#1999)
+
+## v4.2 (2024-01-19)
+
+- Tag release for legacy iRODS v4.2 image
diff --git a/README.md b/README.md
@@ -1,25 +1,21 @@
 # Dockerized iRODS
 
-This repository contains the necessary files to build an iRODS Docker image based on Ubuntu 18.04.
+This repository contains the necessary files to build an iRODS Docker image based on Ubuntu 20.04.
 The code is based on [hurngchunlee/docker-irods](https://github.com/hurngchunlee/docker-irods).
 
 The image contains features specific to our [SODAR](https://github.com/bihealth/sodar-server) system, but using them is optional and the image also works as a generic iRODS server.
 
-## Building
+Images are built and tagged for a specific iRODS release. The most recent build is tested to be compatible with iRODS version `4.3.3`. Our goal is to keep up with the most recent major release of iRODS. Updates for older major versions will not be made.
 
-```bash
-$ cd docker
-$ docker build .
-```
+**NOTE:** Images built for iRODS v4.3.x are **not** compatible with iRODS v4.2 or below. See below for instructions on upgrading from an older iRODS v4.2 build of this image.
 
 ## Data Persistency
 
 Each container exposes volumes for data persistency. The list of volumes are provided in the table below:
 
-| path in container               | usage                         |
-|---------------------------------|-------------------------------|
-| /etc/irods                      | resource server configuration |
-| /var/lib/irods/iRODS/server/log | resource server log           |
+| path in container               | usage                           |
+|---------------------------------|---------------------------------|
+| /etc/irods                      | Server configuration            |
 
 For iRODS services, the setup script (`/var/lib/irods/scripts/setup_irods.py`) is only executed when the file `/etc/irods/.provisioned` is not present.
 The file `/etc/irods/.provisioned` is created when the setup script is executed successfully.
@@ -40,7 +36,8 @@ iRODS can be run in either "provider" mode, which installs an iCAT catalogue ser
 
 | Variable name                    | Default Value                    | Role       |
 |----------------------------------|----------------------------------|------------|
-| IRODS_PKG_VERSION                | 4.2.8-1                          | both       |
+| IRODS_PKG_VERSION                | 4.3.3                            | both       |
+| IRODS_PYTHON_RULE_ENGINE_VERSION | 4.3.3.0-0+4.3.3                  | both       |
 | IRODS_ROLE                       | provider                         | both       |
 | IRODS_HOST_NAME                  | localhost                        | both       |
 | IRODS_SERVICE_ACCOUNT_USER       | irods                            | both       |
@@ -78,15 +75,32 @@ iRODS can be run in either "provider" mode, which installs an iCAT catalogue ser
 
 ## SSSD Support
 
-In addition to the base image, we provide the images `${VERSION}-sssd` (e.g., `4.2.11-1-sssd`) which have SSSD installed.
+In addition to the base image, we provide the images `${VERSION}-sssd` (e.g., `4.3.3-1-sssd`) which have SSSD installed.
 You will have to share `/var/lib/sss` between the SSSD container and iRODS so both containers can communicate.
 
 In our installations, we run [bihealth/sssd-docker](https://github.com/bihealth/sssd-docker) in a second container.
 
+## Upgrading From iRODS 4.2
+
+**TODO**
+
 ## Troubleshooting
 
 A previous version of this image was built on CentOS7 instead of Ubuntu. If updating or redeploying an existing installation, you may encounter the following error connecting to the iRODS database: `[unixODBC][Driver Manager]Data source name not found, and no default driver specified`
 
 To fix this, first edit the file `/etc/irods/server_config.json`. Find the variable `db_odbc_driver` and change its value from `PostgreSQL` to `PostgreSQL Unicode`.
 
 Next, do the same modification for the environment variable `IRODS_ODBC_DRIVER`. After restarting the image, iRODS should work normally.
+
+## Building (for Developers)
+
+To build the image, use the following command:
+
+```
+bash
+$ IRODS_PKG_VERSION=x.x.x IRODS_PYTHON_RULE_ENGINE_VERSION=y.y.y BUILD_VERSION=z ./build.sh
+```
+
+Releases and images are tagged with the iRODS server version followed by the image build version. This means that e.g. the initial release for iRODS `4.3.3` will be tagged as `4.3.3-1`. Fixes or improvements to that release would then be published as `4.3.3-2`.
+
+Note that if you are providing a non-default iRODS version, you will also have to provide the `irods-rule-engine-plugin-python` version number with the `IRODS_PYTHON_RULE_ENGINE_VERSION` env var. This package does not follow the same versioning conventions as the main iRODS packages. The value is expected to be the full version name *without* the `~focal` suffix. You can find the available versions e.g. by running `apt-cache madison irods-rule-engine-plugin-python`.
diff --git a/build.sh b/build.sh
@@ -1,18 +1,24 @@
 #!/bin/bash
 
 export REPO=ghcr.io/bihealth/irods-docker
-export IRODS_PKG_VERSION=${IRODS_PKG_VERSION-4.2.11-1}
+export IRODS_PKG_VERSION=${IRODS_PKG_VERSION-4.3.3}
+export IRODS_PYTHON_RULE_ENGINE_VERSION=${IRODS_PYTHON_RULE_ENGINE_VERSION-4.3.3.0-0+4.3.3}
+export BUILD_VERSION=${BUILD_VERSION-1}
 
 docker build \
-    -t "${REPO}:${IRODS_PKG_VERSION}" \
+    -t "${REPO}:${IRODS_PKG_VERSION}-${BUILD_VERSION}" \
+    --build-arg IRODS_PKG_VERSION=${IRODS_PKG_VERSION} \
+    --build-arg IRODS_PYTHON_RULE_ENGINE_VERSION=${IRODS_PYTHON_RULE_ENGINE_VERSION} \
     --target main \
     docker
 
 docker build \
-    -t "${REPO}:${IRODS_PKG_VERSION}-sssd" \
+    -t "${REPO}:${IRODS_PKG_VERSION}-${BUILD_VERSION}-sssd" \
+    --build-arg IRODS_PKG_VERSION=${IRODS_PKG_VERSION} \
+    --build-arg IRODS_PYTHON_RULE_ENGINE_VERSION=${IRODS_PYTHON_RULE_ENGINE_VERSION} \
     --target sssd \
     docker
 
 echo "Now do:"
-echo "docker push ${REPO}:${IRODS_PKG_VERSION}"
-echo "docker push ${REPO}:${IRODS_PKG_VERSION}-sssd"
+echo "docker push ${REPO}:${IRODS_PKG_VERSION}-${BUILD_VERSION}"
+echo "docker push ${REPO}:${IRODS_PKG_VERSION}-${BUILD_VERSION}-sssd"
diff --git a/docker/Dockerfile b/docker/Dockerfile
@@ -1,15 +1,18 @@
 #
 # Stage: first / main
 #
-FROM ubuntu:18.04 as main
+FROM ubuntu:20.04 AS main
 
 LABEL org.opencontainers.image.authors="Manuel Holtgrewe <manuel.holtgrewe@bih-charite.de>, Mikko Nieminen <mikko.nieminen@bih-charite.de>"
 LABEL org.opencontainers.image.source https://github.com/bihealth/irods-docker
 
 ARG DEBIAN_FRONTEND=noninteractive
+ARG IRODS_PKG_VERSION="4.3.3"
+ARG IRODS_PKG_SUFFIX="-0~focal"
+ARG IRODS_PYTHON_RULE_ENGINE_VERSION="4.3.3.0-0+4.3.3"
 
 # Environment variables for container runtime
-ENV IRODS_PKG_VERSION=4.2.11-1 \
+ENV IRODS_PKG_VERSION=$IRODS_PKG_VERSION \
     IRODS_ROLE=provider \
     IRODS_HOST_NAME=localhost \
     IRODS_SERVICE_ACCOUNT_USER=irods \
@@ -52,44 +55,62 @@ RUN chmod +x /usr/local/bin/wait
 
 # Install general dependencies
 RUN apt-get update && apt-get install -y apt-utils
-RUN apt-get install -y python python-pip python-dev sudo vim wget netcat lsb-release
+RUN apt-get install -y python3 python3-dev python3-distro python3-pip sudo vim wget netcat rsyslog
+# lsb-release g++ gnupg2
 
 # Install database dependencies
 RUN apt-get install -y unixodbc unixodbc-dev odbc-postgresql
 
+# Install iRODS Python dependencies
+RUN pip3 install pyodbc
+
 # Install iRODS
 RUN wget -qO - https://packages.irods.org/irods-signing-key.asc | sudo apt-key add - \
-  && echo "deb [arch=amd64] https://packages.irods.org/apt/ $(lsb_release -sc) main" | sudo tee /etc/apt/sources.list.d/renci-irods.list \
+  && echo "deb [arch=amd64] https://packages.irods.org/apt/ focal main" | sudo tee /etc/apt/sources.list.d/renci-irods.list \
   && apt-get update
-RUN apt-get install -y irods-server=${IRODS_PKG_VERSION}~bionic \
-    irods-database-plugin-postgres=${IRODS_PKG_VERSION}~bionic \
-    irods-dev=${IRODS_PKG_VERSION}~bionic \
-    irods-rule-engine-plugin-python
+RUN apt-get install -y irods-runtime=${IRODS_PKG_VERSION}${IRODS_PKG_SUFFIX} \
+    irods-server=${IRODS_PKG_VERSION}${IRODS_PKG_SUFFIX} \
+    irods-database-plugin-postgres=${IRODS_PKG_VERSION}${IRODS_PKG_SUFFIX} \
+    irods-icommands=${IRODS_PKG_VERSION}${IRODS_PKG_SUFFIX} \
+    irods-dev=${IRODS_PKG_VERSION}${IRODS_PKG_SUFFIX} \
+    irods-rule-engine-plugin-python=${IRODS_PYTHON_RULE_ENGINE_VERSION}~focal
 RUN useradd -d /var/lib/irods ${IRODS_SERVICE_ACCOUNT_USER}
 
 # Copy rule-engine installer
 COPY files/irods_python-re_installer.py /
 RUN chmod +x /irods_python-re_installer.py
 
 # Install j2cli for templating
-RUN apt-get install -y python-jinja2 python-yaml
-RUN pip install j2cli
+RUN apt-get install -y python3-jinja2 python3-yaml
+RUN pip3 install j2cli
 
 # Install Python PAM support
-RUN apt-get install libpam-python pamtester
+RUN apt-get install -y libpam-python pamtester
+# NOTE: Python2 needed for custom PAM module
+RUN apt-get install -y python python-dev
+RUN wget https://bootstrap.pypa.io/pip/2.7/get-pip.py
+RUN python2 get-pip.py
+RUN pip install requests
 
 # Copy scripts and templates
 COPY docker-entrypoint.sh files/irods_login.sh \
      templates/core.py.template templates/unattended_config.json.j2 \
      templates/irods.pam.j2 files/j2-filters.py templates/pam_sodar.py.j2 /
 RUN chmod +x /docker-entrypoint.sh /irods_login.sh
 
+# Set up logging
+COPY files/irods_syslog.conf /etc/rsyslog.d/00-irods.conf
+COPY files/irods.logrotate /etc/logrotate.d/irods
+RUN chmod 0644 /etc/logrotate.d/irods
+RUN mkdir -p /var/log/irods
+RUN chown syslog:adm /var/log/irods
+
 # Create iRODS vault dir
 RUN mkdir -p $IRODS_RESOURCE_DIRECTORY
 RUN chown -cR $IRODS_SERVICE_ACCOUNT_GROUP:$IRODS_SERVICE_ACCOUNT_USER $IRODS_RESOURCE_DIRECTORY
 
 # Data volumes
-VOLUME "/etc/irods" "/var/lib/irods/iRODS/server/log"
+VOLUME "/etc/irods"
 
 # Network ports
 EXPOSE 4321 $IRODS_ZONE_PORT $IRODS_CONTROL_PLANE_PORT $IRODS_DATA_PORT_RANGE_START-$IRODS_DATA_PORT_RANGE_END

diff --git a/docker/docker-entrypoint.sh b/docker/docker-entrypoint.sh
@@ -1,6 +1,7 @@
 #!/bin/bash
 
 NO_WAIT=${NO_WAIT-0}
+export PGPASSWORD=$IRODS_ICAT_DBPASS
 set -euo pipefail
 
 if [[ "$1" == "irods-start" ]]; then
@@ -11,14 +12,16 @@ if [[ "$1" == "irods-start" ]]; then
 
     chmod a+x /var/lib/irods/irodsctl
     chown -cR $IRODS_SERVICE_ACCOUNT_GROUP:$IRODS_SERVICE_ACCOUNT_USER /etc/irods
+    sed -i '/imklog/s/^/#/' /etc/rsyslog.conf
+    chown syslog:adm /var/log/irods
+    /etc/init.d/rsyslog start
 
     echo "iRODS server role: $IRODS_ROLE"
 
     if [[ "$IRODS_ROLE" == "provider" ]] && [[ "$NO_WAIT" -ne 1 ]]; then
         echo "Waiting for postgres.."
         export WAIT_HOSTS=${WAIT_HOSTS-${IRODS_ICAT_DBSERVER}:${IRODS_ICAT_DBPORT}}
         /usr/local/bin/wait
-        PGPASSWORD=$IRODS_ICAT_DBPASS
         PSQL="pg_isready -h $IRODS_ICAT_DBSERVER -p $IRODS_ICAT_DBPORT"
         fi
     fi
@@ -36,29 +39,43 @@ if [[ "$1" == "irods-start" ]]; then
             cp /etc/irods/.odbc.ini /var/lib/irods/.odbc.ini
         fi
 
+        if [ -f /etc/irods/version.json ]; then
+            cp -f /etc/irods/version.json /var/lib/irods/version.json
+        fi
+
     else
 
         echo "Provisioning iRODS.."
 
         if [[ "$IRODS_ROLE" == "provider" ]]; then
-            echo "Pre-create database if necessary"
-            echo $IRODS_ICAT_DBPASS \
-            | createdb -h $IRODS_ICAT_DBSERVER -p $IRODS_ICAT_DBPORT -U $IRODS_ICAT_DBUSER -W $IRODS_ICAT_DBNAME
+
+            if [ "$( psql -h $IRODS_ICAT_DBSERVER -p $IRODS_ICAT_DBPORT -U $IRODS_ICAT_DBUSER \
+                -XtAc "SELECT 1 FROM pg_database WHERE datname='$IRODS_ICAT_DBNAME'" )" = '1' ]
+            then
+                echo "iCAT database already exists, skipping creation"
+            else
+                echo "Create iCAT database"
+                createdb -h $IRODS_ICAT_DBSERVER -p $IRODS_ICAT_DBPORT -U $IRODS_ICAT_DBUSER -W $IRODS_ICAT_DBNAME
+            fi
+
         fi
 
         echo "Set up unattended configuration file"
         j2 -o /unattended_config.json --undefined --filters=j2-filters.py unattended_config.json.j2
+        # DEBUG
+        # cat /unattended_config.json
 
         echo "Perform iRODS setup"
-        python /var/lib/irods/scripts/setup_irods.py --json_configuration_file=/unattended_config.json
+        python3 /var/lib/irods/scripts/setup_irods.py --json_configuration_file=/unattended_config.json
 
         cp /var/lib/irods/.irods/irods_environment.json /etc/irods/irods_environment.json
         cp /var/lib/irods/.odbc.ini /etc/irods/.odbc.ini
+        cp -f /var/lib/irods/version.json /etc/irods/version.json
 
         # Enable the python rule engine
         if [ -f /irods_python-re_installer.py ]; then
             echo "Enable python rule engine"
-            ./irods_python-re_installer.py
+            python3 ./irods_python-re_installer.py
         fi
 
         touch /etc/irods/.provisioned

diff --git a/docker/files/irods.logrotate b/docker/files/irods.logrotate
@@ -0,0 +1,9 @@
+/var/log/irods/irods.log {
+    weekly
+    rotate 26
+    copytruncate
+    dateext
+    notifempty
+    missingok
+    su root root
+}
diff --git a/docker/files/irods_syslog.conf b/docker/files/irods_syslog.conf
@@ -0,0 +1,8 @@
+$FileCreateMode 0644
+$DirCreateMode 0755
+$Umask 0000
+$template irods_format,"%msg%\n"
+:programname,startswith,"irodsServer" /var/log/irods/irods.log;irods_format
+& stop
+:programname,startswith,"irodsDelayServer" /var/log/irods/irods.log;irods_format
+& stop
diff --git a/docker/templates/pam_sodar.py.j2 b/docker/templates/pam_sodar.py.j2
@@ -5,6 +5,9 @@ LDAP/AD server is not available.
 __author__ = 'Mikko Nieminen'
 
 import os
+import site
+# HACK to allow libpam-python to find 3rd party packages
+site.addsitedir('/usr/local/lib/python2.7/site-packages')
 import requests
 
 
@@ -17,7 +20,7 @@ def pam_sm_authenticate(pamh, flags, argv):
     sodar_host = '{{ IRODS_SODAR_API_HOST }}'
     url = sodar_host + '/irodsbackend/api/auth'
 
-    response = requests.post(url, auth=(pamh.user, a))
+    response = requests.get(url, auth=(pamh.user, a))
     if response.status_code == 200:
         return pamh.PAM_SUCCESS
     return pamh.PAM_AUTH_ERR