Skip to content

DIRAC v7r2

Jump to bottom Edit New page
Chris Burr edited this page Nov 17, 2021 · 49 revisions

Rename of scripts

For server installations: the runit scripts have to be modified to run dirac-service instead of $DIRAC/DIRAC/Core/scripts/dirac-service.py. Same for dirac-agent and dirac-executor. So, login in each of your server installations, sudo su dirac and run:

sed -i.bak -E 's@python \$DIRAC/DIRAC/Core/scripts/dirac[-_]([a-z]+)\.py@dirac-\1@g' /opt/dirac/runit/*/*/run

⚠️ IMPORTANT: HTTPS and WebApp services will also need to be changed from exec python /path/to/something.py to exec something.

NB : this is optional in v7r2 but will be required before moving to v7r3.

Python3 based clients

This version adds support for Python 3 based clients, using DIRACOS2 and python pip. The installation instructions can be found in DIRAC main README file. The suggestion is to start using python 3 based clients while v7r2 (7.2) is in production. Python 3 server comes in version 7.3.

Code structure, and moving towards python3

If you have a DIRAC extension, and want to run python3 clients of your DIRAC extension, there are a few things to be done on your side:

  • The structure with src/PROJECTNAME/code... should be adopted
  • A setup.py and/or setup.cfg file(s) should be written. See examples in LHCbDIRAC code
  • For Vanilla projects this affects COMDIRAC, and VMDIRAC, while WebAppDIRAC is only server, so no need to update immediately.

ElasticSearch versions supported

ElasticSearch version 7.x is now supported. The support for version 5 or earlier could not be assured.

Move VOPolicy/InputDataModule to Operations

In order to support multi VO and make the InputData module work with certificate needs to be moved to Operations section (https://github.com/DIRACGrid/DIRAC/pull/4339):

/DIRAC/VOPolicy/InputDataModule -> /DIRAC/Operations/InputDataPolicy/InputDataModule

also since https://github.com/DIRACGrid/DIRAC/pull/5172:

/DIRAC/VOPolicy/SoftwareDistModule -> /DIRAC/Operations/SoftwareDistModule

yes, into the base section.

Removal of PyGSI

The home made wrapper of openssl (pyGSI) has been finally fully removed in favor of the more standard M2Crypto library. It was first made available in v7r0 and was the default in v7r1.

M2Crypto (or any standard tool that respects TLS..) is stricter than PyGSI. So you may need to adapt your environment a bit. Here are a few hints:

  • SAN in your certificates: if you are contacting a machine using its aliases, make sure that all the aliases are in the SubjectAlternativeName (SAN) field of the certificates
  • FQDN in the configuration: SAN normally contains only FQDN, so make sure you use the FQDN in the CS as well (e.g. mymachine.cern.ch and not mymachine)
  • ComponentInstaller screwed: like any change you do on your hosts, the ComponentInstaller will duplicate the entry. So if you change the CS to put FQDN, the machine will appear twice.

Framework

MySQL

Within PR https://github.com/DIRACGrid/DIRAC/pull/4956 we renamed a few columns of MySQL tables, to avoid conflicts with reserved keywords in MySQL 8. Specifically:

InstalledComponentsDB.Component.System -> InstalledComponentsDB.Component.DIRACSystem
InstalledComponentsDB.Component.Module -> InstalledComponentsDB.Component.DIRACModule

ALTER TABLE `Components` RENAME COLUMN `System` TO `DIRACSystem`;
ALTER TABLE `Components` RENAME COLUMN `Module` TO `DIRACModule`;

For mysql version 13 the commands to update the database are:

ALTER TABLE Components CHANGE System DIRACSystem varchar(32) NOT NULL;
ALTER TABLE Components CHANGE Module DIRACModule varchar(32) NOT NULL;

Additionally, #5092 lengthens the DIRACVersion column to account for the potentially longer version numbers in Python 3 based installations:

USE InstalledComponentsDB;
ALTER TABLE HostLogging MODIFY COLUMN DIRACVersion varchar(64);

gLogger level mechanism

Within https://github.com/DIRACGrid/DIRAC/pull/4610 we changed the way the logger level is set to make it closer to the standard logging library. The level is currently managed in a strange way:

  • the logger level is always debug
  • Logging.setLevel() just change the level of the Backends (so of the handlers) attached to the Logging

This was done in order to make sure that the centralized logging could not be disabled: all logs from all Logging objects go to a MessageQueue for instance. In practice, subloggers don't have any Backend and just propagate the logs to LoggingRoot that spreads the logs across its Backends. Thus, Logging.setLevel() has no effect from a sublogger.

From v7r2 the Logging level is attached to the logger level to better respect the standard mechanism:

  • Logging.setLevel() works from every sublogger and change the level of its underlying logger
  • Backend level can still be changed from the configuration to only allow log records from a certain level

This means that the centralized logging can only get logs that respects the following condition: log record level > logger level:

LogBackends
{
  Stdout
  {
    LogLevel='INFO'
  }
}

Keep in mind that if you change the Backend level, you will not be able to get logs below this level, even if you change the Logging level.

New ComponentSupervisionAgent since v7r2p8

There is a new agent with some functionality to supervise the state of services, agents, and executors: https://dirac.readthedocs.io/en/latest/CodeDocumentation/FrameworkSystem/Agent/ComponentSupervisionAgent.html

Monitoring

Note (July 26, 2021): This is not yet ready for use!

- Component monitoring type should now be reliable - NEW: RMSMonitoring In order to enable RMSMonitoring we need to set value of EnableActivityMonitoring flag to yes/true inside /Operations/Defaults or /Systems/RequestManagement/<Instance>/Agents/RequestExecutingAgent of the cfg file.

TransformationSystem

You can optionally modify Transformations table to store GroupSize as FLOAT (PR):

ALTER TABLE `Transformations` MODIFY `GroupSize` FLOAT NOT NULL DEFAULT 1;

WorkloadManagement

Changes in pilot files synchronization

The PR 4874 did two things:

  • removed the ConfigurationServer hook for updating the pilot files.
  • introduced a new agent and a script for helping synchronizing the pilot files, offering a way to solve issue 4697. The agent is called PilotSyncAgent and can be installed as all other agents using the dirac-admin-sysadmin-cli. Please refer to the agent documentation for a complete info on how to use it.
  • A script is also provided with the same functionalities of the agent.

New JobDB option CompressJDLs

From #4367 To activate, add CompressJDLs=True option in /Systems/WorkloadManagement/<setup>/DB/JobDB

Elastic Job Parameters on ElasticSearch

Following a previous "tech preview" annouce, you can now safely use ElasticSearch for Job Parameters: see doc

StatesMonitoringAgent -> StatesAccountingAgent

PR https://github.com/DIRACGrid/DIRAC/pull/4905 adds a new flag to the StatesAccountingAgent, called Backends (BACKENDS WITH S!!!!), which can take the value of Accounting (which is the default), or Monitoring or Accounting,Monitoring. If Monitoring is in the backends list, then the StatesAccountingAgent replaces the StatesMonitoringAgent by committing to (also) the ElasticSearch backend of the Monitoring system.

ResourceStatus System (RSS)

Run the following MySQL commands:

use ResourceManagementDB;
ALTER TABLE `DowntimeCache` MODIFY `DowntimeID` VARCHAR(127);

Web configuration

The file etc/web.cfg is not read anymore: before upgrading/installing, make sure that its content is added top the CS. Starting with this version, a description of the organization of the simple server file to serve the static files, this is not necessarily an additional option. For this, in addition to nginx itself, you need to install the web dev plug-in, what is described in the WebAppDIRAC installation instructions. Note that this nginx configuration can also be installed separately from the web portal and used as an additional file server.

DataManagement

Changes for DFC schema

This is only if you use the LHCb managers (FileManagerPS and DirectoryClosure). Following #5057, 4 procedures need to be recreated:

ps_get_dir_logical_size
ps_calculate_dir_logical_size
ps_get_dir_physical_size
ps_calculate_dir_physical_size

To do so, basically connect to the DB and execute this whole block https://github.com/DIRACGrid/DIRAC/blob/4176025445ff0b79d5b3aecf862ecb47139e0756/src/DIRAC/DataManagementSystem/DB/FileCatalogWithFkAndPsDB.sql#L1647-L1801

New Status for FTS3 Files and Jobs

In order to use the Archiving feature of FTS

alter table Files CHANGE status status enum('New','Submitted','Ready','Active','Finished','Canceled','Staging','Failed','Defunct','Started','Not_used','Archiving') DEFAULT 'New';
alter table Jobs CHANGE status status enum('Submitted','Ready','Active','Finished','Canceled','Failed','Finisheddirty','Staging','Archiving') DEFAULT 'Submitted';

For more details, see https://github.com/DIRACGrid/DIRAC/pull/5384

Add a custom footer
Add a custom sidebar
Clone this wiki locally