DIRAC v7r2

Python3 based clients

With this version, there is a experimental support for Python 3 based, using DIRACOS2 and python pip. The installation instructions can be found in DIRAC main README file.

Code structure, and moving towards python3

If you have a DIRAC extension, and want to run python3 clients of your DIRAC extension, there are a few things to be done on your side:

• The structure with src/PROJECTNAME/code... should be adopted
• A setup.py and/or setup.cfg file(s) should be written. See examples in LHCbDIRAC devel code
• For Vanilla projects this affects COMDIRAC, and VMDIRAC, while WebAppDIRAC is only server, so no need to update immediately.

ElasticSearch versions supported

ElasticSearch version 7.x is now supported. The support for version 5 or earlier could not be assured.

Move VOPolicy/InputDataModule to Operations

In order to support multi VO and make the InputData module work with certificate needs to be moved to Operations section:

/DIRAC/VOPolicy/InputDataModule -> /DIRAC/Operations/InputDataPolicy

Removal of PyGSI

The home made wrapper of openssl (pyGSI) has been finally fully removed in favor of the more standard M2Crypto library. It was first made available in v7r0 and was the default in v7r1.

M2Crypto (or any standard tool that respects TLS..) is stricter than PyGSI. So you may need to adapt your environment a bit. Here are a few hints:

• SAN in your certificates: if you are contacting a machine using its aliases, make sure that all the aliases are in the SubjectAlternativeName (SAN) field of the certificates
• FQDN in the configuration: SAN normally contains only FQDN, so make sure you use the FQDN in the CS as well (e.g. mymachine.cern.ch and not mymachine)
• ComponentInstaller screwed: like any change you do on your hosts, the ComponentInstaller will duplicate the entry. So if you change the CS to put FQDN, the machine will appear twice.

Framework

Within PR https://github.com/DIRACGrid/DIRAC/pull/4956 we renamed a few columns of MySQL tables, to avoid conflicts with reserved keywords in MySQL 8. Specifically:

InstalledComponentsDB.Component.System -> InstalledComponentsDB.Component.DIRACSystem
InstalledComponentsDB.Component.Module -> InstalledComponentsDB.Component.DIRACModule

ALTER TABLE `Components` RENAME COLUMN `System` TO `DIRACSystem`
ALTER TABLE `Components` RENAME COLUMN `Module` TO `DIRACModule`

Additionally, #5092 lengthens the DIRACVersion column to account for the potentially longer version numbers in Python 3 based installations:

USE InstalledComponentsDB;
ALTER TABLE HostLogging MODIFY COLUMN DIRACVersion varchar(64);

Monitoring

• Component monitoring type should now be reliable
• NEW: RMSMonitoring In order to enable RMSMonitoring we need to set value of EnableActivityMonitoring flag to yes/true inside /Operations/Defaults or /Systems/RequestManagement/<Instance>/Agents/RequestExecutingAgent of the cfg file.

WorkloadManagement

Changes in pilot files synchronization

The PR 4874 did two things:

• removed the ConfigurationServer hook for updating the pilot files.
• introduced a new agent and a script for helping synchronizing the pilot files, offering a way to solve issue 4697. The agent is called PilotSyncAgent and can be installed as all other agents using the dirac-admin-sysadmin-cli. Please refer to the agent documentation for a complete info on how to use it.
• A script is also provided with the same functionalities of the agent.

New JobDB option CompressJDLs

From #4367 To activate, add CompressJDLs=True option in /Systems/WorkloadManagement/<setup>/DB/JobDB

Elastic Job Parameters on ElasticSearch

Following a previous "tech preview" annouce, you can now safely use ElasticSearch for Job Parameters: see doc

StatesMonitoringAgent -> StatesAccountingAgent

PR https://github.com/DIRACGrid/DIRAC/pull/4905 adds a new flag to the StatesAccountingAgent, called Backend, which can take the value of Accounting (which is the default), or Monitoring or Accounting,Monitoring. If Monitoring is in the backends list, then the StatesAccountingAgent replaces the StatesMonitoringAgent by committing to (also) the ElasticSearch backend of the Monitoring system.

ResourceStatus System (RSS)

Run the following MySQL commands:

use ResourceManagementDB;
ALTER TABLE `DowntimeCache` MODIFY `DowntimeID` VARCHAR(127);

Web configuration

The file etc/web.cfg is not read anymore: before upgrading/installing, make sure that its content is added top the CS. Starting with this version, a description of the organization of the simple server file to serve the static files, this is not necessarily an additional option. For this, in addition to nginx itself, you need to install the web dev plug-in, what is described in the WebAppDIRAC installation instructions. Note that this nginx configuration can also be installed separately from the web portal and used as an additional file server.

Rename of scripts

The python script implementing the commands in DIRAC have been renamed to contain _ instead of -. The actual user facing commands are unaffected (i.e. dirac-proxy-init is now implemented in dirac_proxy_init.py). This requires the runit scripts to be modified to run dirac_service.py instead of dirac-service.py.

find -L /opt/dirac/startup/ -name run -exec sed -i.bak -E 's|dirac-([a-z]+).py|dirac_\1.py|g' {} \;

This is optional in v7r2 but will be required before moving to v7r3.

DataManagement

Changes for DFC schema

This is only if you use the LHCb managers (FileManagerPS and DirectoryClosure). Following https://github.com/DIRACGrid/DIRAC/pull/5057, 4 procedures need to be recreated:

ps_get_dir_logical_size
ps_calculate_dir_logical_size
ps_get_dir_physical_size
ps_calculate_dir_physical_size

To do so, basically connect to the DB and execute this whole block https://github.com/DIRACGrid/DIRAC/blob/4176025445ff0b79d5b3aecf862ecb47139e0756/src/DIRAC/DataManagementSystem/DB/FileCatalogWithFkAndPsDB.sql#L1647-L1801