iRODS: v0.9.8 #504

scimerman · 2021-12-08T19:46:45Z

This is a rought version. The role works runs and can be repeatedly redeployed as a working
iRODS+davrods environment, but missing the

full firewall implementation, and the
postgresql backup and live replication.

Description of major changes:

iRODS

runs on linux machine under username irods (or what is set with variable irods_service_account)
large change is in the roles/irods/templates/unattended_install.json.j2 as
instead of having three separated files (server_config.json.j2,
service_account_environment.j2 and hosts_config.json.j2) it contains all
the paramateres in one file. This is because the file was generated according
to the instructions in the role/README.md - that is, it was exported with the
used of command izonereport on the working iCAT server.
I believe this should be the recommended way anyway, instead of potentially
outdated online templates.
add the irods_fix.conf so that the longer transfers do not get timeout
add dummy /etc/irods/.s3auth - so that potential remote S3 resources even work
Postgres: pg_hba.conf allows the local users from local network to access to
database
iRODS installation starts only if iCAT database does not exists OR if it is
empty
reruning the installation script DOES NOT reinstall the iRODS or format the
database, but the installation script does try to restart service, and so this
is tried to be avoided

Davrods

Davrods templates folder contains the downloaded latest Davrods version, but
we made changes in the

on linux machine it deploys Docker under username davrods (or what is set with variable davrods_docker_user)
Dockerfile
- add the SSL options - installed the Apache mod ssl extension
- add copy the key, chain and dhparam files into Docker
- ports: closed port 80, opened instead 443
.env
- add: versions of ENV_DAVRODS_IRODS_VERSION and ENV_DAVRODS_VERSION parameters
docker-compose.yml
- ports: closed 80, opened 443
- made sure the docker is always restated (tested: reboot and process killed)
davrods-vhost.conf
- the existing .conf was working over the normal http communication
- I added the options to use the SSL certificates that are installed with ^
  Dockerfile
- the problematic (latest version of Centos 7.9 Apache does not support the
  use of dhparam file, so it is appended at the end of the .cer file)
- the rest of the file uses variables from irods.yml and/or from defaults to
  set up the davrods parameters
- the interesting part is the variable DavRodsServer, which sets the iRODS
  address for the Davrods. That address is using the internal (f.e.
  10.10.2.123) address for the communication.

deploy-os_servers.yml

Opened the port 5432 for the PostgreSQL (for the use of live replication).

Updated README.md

mostly the TO-DO's

Secrets

created new and replaced the od (default) passwords with more secure ones

iRODS: readme update - added PN mail to todo iRODS: formatting fix - readme Update README.md iRODS: formatting - readme iRODS: edited readme

…e Dockerfile

pneerincx · 2021-12-10T09:28:29Z

deploy-os_servers.yml

@@ -326,6 +326,16 @@
 remote_ip_prefix: 0.0.0.0/0
 wait: true
 timeout: "{{ openstack_api_timeout }}"
+ - name: "Add rule to {{ slurm_cluster_name }}_irods security group: allow PostgreSQL inbound on port 5432."


Maybe add to the ToDo list that like for iRODS port 1247, this port for PostgreSQL should be restricted to specific machines as opposed to allowing inbound traffic from anywhere.

Yes, for the SQL, we should have an IP whitelist yes. Added to the TODO.

For the iRODS 1247 on the other hand: we haven't yet agreed if we really want to limit the port 1247 to SURF side server only, as there was once mentioned that we might allow users to directly access iRODS over this protocol as well (some clients support it) - instead of going through Davrods.

pneerincx · 2021-12-10T09:31:28Z

roles/irods/tasks/icat.yml

- - irods-database-plugin-postgres
+ - irods-server
+ - irods-database-plugin-postgres
+ - irods-resource-plugin-s3-4.2.10.0-1


Is it necessary to hard-code the version number for this S3 plugin?

Correct. Tested without - works. Removed in new commit.

pneerincx · 2021-12-10T09:33:58Z

roles/irods/tasks/icat.yml

 register: create_unattended_installation_json
+ become: true
+
+# - name: Check if {{ irods_db_name }} database is available


Do we need the commented code or can this be deleted?

Was there, as this part was only recently changed. And was for the potential fallback. Removed now.

pneerincx · 2021-12-10T09:36:30Z

roles/irods/tasks/icat.yml

+
+# Start/stop/restart service call fails if with error if already in that state
+# so it's best if done via user call (as this is what the service does anyways)
+- name: Restart iRODS server manually via {{ irods_service_account }}


Services should only be restarted if necessary. E.g. when some configuration has changed or software got updated. This should be moved to a handler.

✔️ The new commit has (tested successfully) handlers.

Why the system service was not used: (the explanation for the rest):

Sysvinit service cannot be used, as ansible calls support only start, stop, restarted and reloaded:

the start does not work - even when irods is shut down, it does not start it up (repeatably) and subsequent Davrods role fails

the restarted repeatably fails upon new installation when service is still stopped (see output below, but in short: service uses lock file /var/lock/subsys/irods that cannot remove it, as it does not exists). I am certain this is a bug in a /etc/init.d/irods

since the stop and reloaded (which does not help, nor is implemented in irods service definition file) this means the native sysvinit service call cannot be used.

My implementation avoids use of service call and directly restarts the server in the same way as the service itself would (I looked into the /etc/init.d/irods, it is basically just /var/lib/irods/irodsctl script). Direct call does not fail the role, and we do not need to add into ansible the wrappers to catch errors. It also only fails when there is an actual error with irods server startup.

Problem with system service restarted call:

TASK [irods : Enable and iRODS service] ********************************************************************************************************************* fatal: [tunnel+irods-test]: FAILED! => changed=false msg: 'Failed to restart service: irods' rc: 1 stderr: |- [1;33mNo iRODS servers running.[0m rm: cannot remove ‘/var/lock/subsys/irods’: No such file or directory stderr_lines: <omitted> stdout: '' stdout_lines: <omitted>

This bug can be also resolved by changing the /etc/init.d/irods line:
if [ "$DETECTEDOS" = "RedHatCompatible" -o "$DETECTEDOS" = "SuSE" ]; then
into
if [ "$DETECTEDOS" = "RedHatCompatible" -o "$DETECTEDOS" = "SuSE" ] && [ -e /var/lock/subsys/irods ]; then

Bug confirmed, going to be fixed in upcoming versions: irods/irods#6053

pneerincx · 2021-12-10T09:38:00Z

roles/irods/tasks/pgsql.yml


- name: Creating irods sql user and granting privileges
+- name: Restarting PGSQL and making sure it is enabled


Use a handler to restart services only when required.

pneerincx · 2021-12-10T09:42:26Z

roles/irods/tasks/icat.yml

+
+- name: Fix the s3 authentication by creating dummy s3auth file
+ ansible.builtin.copy:
+ content: |


When content is specified like this, we do not need the roles/irods/files/s3auth file anymore, right?

Correct. Was doing it one way, then another. This is not needed anymore and will remove it ✅

pneerincx

Nice work! Some small comments/questions. See inline comments for details.

pneerincx · 2021-12-16T15:48:36Z

roles/irods/tasks/icat.yml


 - name: If missing, build 4096 bit DHparam /etc/irods/{{ irods_ssl_dh_params_file }} file (takes several minues ...)
 ansible.builtin.command:
- cmd: /bin/openssl dhparam -2 -out /etc/irods/{{ irods_ssl_dh_params_file }} 4096
+ cmd: /bin/openssl dhparam -2 -out /etc/irods/{{ irods_ssl_dh_params_file }} 2048


Comment still mentions 4096 bits. Why was this downgraded? (Note that 2048 bits will take less time, but does not comply with UMCG encryption policy.)

scimerman added 18 commits December 3, 2021 16:04

added: security group iRODS open 443 port

fa57ead

Merged with iRODS v1

c4aad84

iRODS: v2 working irods & davrods - still WIP

40de7bb

iRODS: add openstack open port for postgresql 5432 for live replication

a52b864

Removing stale developing test template from upstream

81184a0

iRODS: minor bug fixes

c0774ec

iRODS: readme update

c4b157e

iRODS: readme update - added PN mail to todo iRODS: formatting fix - readme Update README.md iRODS: formatting - readme iRODS: edited readme

"iRODS: fix faulty hostname lookup"

ebda4b9

iRODS: split resource variable into local and default

c576103

iRODS: updated README

d373d40

iRODS: split resource variable fix

9354902

iRODS: fixed database hostname to use variable

858275a

fix: removed workign temporary comments

9ab7597

iRODS: set 4 threads in core.re

1df6a11

iRODS: edited irods role README.md

8bfec36

iRODS: Davrods default resource

a49f261

iRODS: estetic fix - port order

a012dad

iRODS: moved the appending of dhparams to the chain from role into th…

911baac

…e Dockerfile

scimerman requested review from pneerincx, Gerbenvandervries and marieke-bijlsma December 8, 2021 19:46

pneerincx reviewed Dec 10, 2021

View reviewed changes

pneerincx requested changes Dec 10, 2021

View reviewed changes

scimerman added 2 commits December 14, 2021 13:16

iRODS: PR fixes added

08270ed

iRODS: davrods make hosting machine independant

7d851ac

iRODS: removed empty spaces and lines

905f130

pneerincx reviewed Dec 16, 2021

View reviewed changes

iRODS: fixed dhparams key size from 2048 to 4096

0ac06cc

pneerincx approved these changes Dec 16, 2021

View reviewed changes

pneerincx merged commit 98939e9 into rug-cit-hpc:develop Dec 16, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

iRODS: v0.9.8 #504

iRODS: v0.9.8 #504

scimerman commented Dec 8, 2021

pneerincx Dec 10, 2021

scimerman Dec 10, 2021

pneerincx Dec 10, 2021

scimerman Dec 10, 2021

pneerincx Dec 10, 2021

scimerman Dec 10, 2021

pneerincx Dec 10, 2021

scimerman Dec 10, 2021

scimerman Dec 14, 2021

pneerincx Dec 10, 2021

pneerincx Dec 10, 2021

scimerman Dec 10, 2021

pneerincx left a comment

pneerincx Dec 16, 2021


		- name: Creating irods sql user and granting privileges
		- name: Restarting PGSQL and making sure it is enabled

iRODS: v0.9.8 #504

iRODS: v0.9.8 #504

Conversation

scimerman commented Dec 8, 2021

iRODS

Davrods

deploy-os_servers.yml

Updated README.md

Secrets

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pneerincx left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment