Skip to content

DIRAC v6r20

Jump to bottom Edit New page
Andre Sailer edited this page Aug 2, 2018 · 37 revisions

Core

This only applies if you have extended DIRAC: in PR https://github.com/DIRACGrid/DIRAC/pull/3675 the DIRAC/Core/Utilities/init.py module has been stripped off all its imports. Re-check your code if you were needing them.

Configuration

The host where the Master CS is running need to have the git binaries installed.

Accounting

Some tables in AccountingDB should be modified, so start with.

USE AccountingDB

The real name of the table depends from your setup, so what follows are just examples (where the setup is "LHCb-Production").

ALTER TABLE `ac_in_LHCb-Production_Job` MODIFY ProcessingType VARCHAR(256) NOT NULL;
ALTER TABLE `ac_key_LHCb-Production_Job_ProcessingType` MODIFY value VARCHAR(256) NOT NULL;

ALTER TABLE `ac_in_LHCb-Production_DataOperation` MODIFY ExecutionSite VARCHAR(256) NOT NULL;
ALTER TABLE `ac_key_LHCb-Production_DataOperation_ExecutionSite` MODIFY value VARCHAR(256) NOT NULL;

Framework

InstalledComponents

use InstalledComponentsDB;
ALTER TABLE HostLogging MODIFY DiskOccupancy varchar(256);

ResourceStatusSystem

The CacheFeederAgent should be restarted first, and this will populate correctly the DBs. (should I stop the others first?

The SpaceTokenOccupancy policy does not exist anymore: the new policy FreeDiskSpacePolicy replace it, and it's valid for all the StorageElement types (srm, xroot, dips, ...) provided that StorageElement(seName).getOccupancy() returns correct results.

This means that you can remove from the CS all mentions to SpaceTokenOccupancy policies, e.g. if you have in CS a section like /Operations/Defaults/ResourceStatus/Policies/SpaceTokenOccupancy it can be removed.

The new policy should instead be added:

Operations
{
  Defaults # or a setup
  {
    ResourceStatus
    {
      Policies
      {
        FreeDiskSpaceTB
        {
          matchParams
          {
            elementType=StorageElement
            statusType=WriteAccess
            element=Resource
          }
        }
      }
    }
  }
}

You should also remove mentions of "SpaceTokenOccupancy" related actions from the /Operations/Defaults/ResourceStatus/PolicyActions/ section.

DataManagementSystem

New FTS3 system

A new system (Agent & Service) meant to replace the old one and add new feature is available. However, both systems (FTS3 and the "old" FTS ones) cannot co-exist. Please refer to the documentation on how to do the switch over http://dirac.readthedocs.io/en/integration/AdministratorGuide/Systems/DataManagement/fts3.html

DB changes to (old) FTSDB

If you don't want to switch to the new FTS3 system, the FTSDB should be changed accordingly to what follows, especially if you want to accommodate the stricter MySQL 5.7 requirements:

USE FTSDB;

ALTER TABLE FTSFile MODIFY Error VARCHAR(2048);
ALTER TABLE FTSJob MODIFY Error VARCHAR(2048);
ALTER TABLE FTSJob MODIFY FailedSize BIGINT;

DFC LHCb Manager

Two stored procedures need to be recreated BEFORE updating to MySQL 5.7: ps_delete_files and ps_delete_replicas_from_file_ids. For this, from the DIRAC/DataManagementSystem/DB/FileCatalogWithFkAndPsDB.sql, for each of the two procedures, copy the whole PL/SQL code chunk, and copy paste it in MySQL

DROP PROCEDURE IF EXISTS [...]
[...]
END //
DELIMITER ;

WorkloadManagementSystem

  • PilotMonitorAgent has been removed. Please de-install it using the SystemAdministration.
  • TaskQueueDirector agent has been removed. Please de-install it using the SystemAdministration.

SiteDirector

NB: VOs with an extension of the SiteDirector should double-check that their extension is still compatible with the changes in the base DIRAC agent.

The SiteDirector agent, a key agent for many DIRAC installation, has been largely updated. It also includes a new flag, the "pilots3" flag for anyone wanting to submit pilots3 via SiteDirectors (pilots 2 stay the default). Please find more info about pilots3 in the official documentation in http://dirac.readthedocs.io/en/integration/AdministratorGuide/Systems/WorkloadManagement/Pilots/Pilots3.html.

PilotAgentsDB:

The PilotRequirements table from PilotAgentsDB can be removed:

USE PilotAgentsDB;
DROP TABLE IF EXISTS PilotRequirements;

TaskQueueDB:

You can apply the following after having installed the release.

use TaskQueueDB;

ALTER TABLE `TaskQueueDB`.`tq_Jobs` CHANGE COLUMN `JobId` `JobId` int(11) UNSIGNED NOT NULL;
ALTER TABLE `TaskQueueDB`.`tq_Jobs` CHANGE COLUMN `TQId` `TQId` int(11) UNSIGNED NOT NULL;
ALTER TABLE `TaskQueueDB`.`tq_TaskQueues` CHANGE COLUMN `TQId` `TQId` int(11) UNSIGNED NOT NULL AUTO_INCREMENT;
ALTER TABLE `TaskQueueDB`.`tq_TQToSites` CHANGE COLUMN `TQId` `TQId` int(11) UNSIGNED NOT NULL;
ALTER TABLE `TaskQueueDB`.`tq_TQToGridCEs` CHANGE COLUMN `TQId` `TQId` int(11) UNSIGNED NOT NULL;
ALTER TABLE `TaskQueueDB`.`tq_TQToGridMiddlewares` CHANGE COLUMN `TQId` `TQId` int(11) UNSIGNED NOT NULL;
ALTER TABLE `TaskQueueDB`.`tq_TQToBannedSites` CHANGE COLUMN `TQId` `TQId` int(11) UNSIGNED NOT NULL;
ALTER TABLE `TaskQueueDB`.`tq_TQToPlatforms` CHANGE COLUMN `TQId` `TQId` int(11) UNSIGNED NOT NULL;
ALTER TABLE `TaskQueueDB`.`tq_TQToPilotTypes` CHANGE COLUMN `TQId` `TQId` int(11) UNSIGNED NOT NULL;
ALTER TABLE `TaskQueueDB`.`tq_TQToSubmitPools` CHANGE COLUMN `TQId` `TQId` int(11) UNSIGNED NOT NULL;
ALTER TABLE `TaskQueueDB`.`tq_TQToJobTypes` CHANGE COLUMN `TQId` `TQId` int(11) UNSIGNED NOT NULL;
ALTER TABLE `TaskQueueDB`.`tq_TQToTags` CHANGE COLUMN `TQId` `TQId` int(11) UNSIGNED NOT NULL;

ALTER TABLE `TaskQueueDB`.`tq_Jobs` ADD FOREIGN KEY (`TQId`) REFERENCES `TaskQueueDB`.`tq_TaskQueues` (`TQId`);

ALTER TABLE `TaskQueueDB`.`tq_TQToSites` ADD PRIMARY KEY (`TQId`, `Value`);
ALTER TABLE `TaskQueueDB`.`tq_TQToSites` ADD FOREIGN KEY (`TQId`) REFERENCES `TaskQueueDB`.`tq_TaskQueues` (`TQId`);
ALTER TABLE `TaskQueueDB`.`tq_TQToGridCEs` ADD PRIMARY KEY (`TQId`, `Value`);
ALTER TABLE `TaskQueueDB`.`tq_TQToGridCEs` ADD FOREIGN KEY (`TQId`) REFERENCES `TaskQueueDB`.`tq_TaskQueues` (`TQId`);
ALTER TABLE `TaskQueueDB`.`tq_TQToGridMiddlewares` ADD PRIMARY KEY (`TQId`, `Value`);
ALTER TABLE `TaskQueueDB`.`tq_TQToGridMiddlewares` ADD FOREIGN KEY (`TQId`) REFERENCES `TaskQueueDB`.`tq_TaskQueues` (`TQId`);
ALTER TABLE `TaskQueueDB`.`tq_TQToBannedSites` ADD PRIMARY KEY (`TQId`, `Value`);
ALTER TABLE `TaskQueueDB`.`tq_TQToBannedSites` ADD FOREIGN KEY (`TQId`) REFERENCES `TaskQueueDB`.`tq_TaskQueues` (`TQId`);
ALTER TABLE `TaskQueueDB`.`tq_TQToPlatforms` ADD PRIMARY KEY (`TQId`, `Value`);
ALTER TABLE `TaskQueueDB`.`tq_TQToPlatforms` ADD FOREIGN KEY (`TQId`) REFERENCES `TaskQueueDB`.`tq_TaskQueues` (`TQId`);
ALTER TABLE `TaskQueueDB`.`tq_TQToPilotTypes` ADD PRIMARY KEY (`TQId`, `Value`);
ALTER TABLE `TaskQueueDB`.`tq_TQToPilotTypes` ADD FOREIGN KEY (`TQId`) REFERENCES `TaskQueueDB`.`tq_TaskQueues` (`TQId`);
ALTER TABLE `TaskQueueDB`.`tq_TQToSubmitPools` ADD PRIMARY KEY (`TQId`, `Value`);
ALTER TABLE `TaskQueueDB`.`tq_TQToSubmitPools` ADD FOREIGN KEY (`TQId`) REFERENCES `TaskQueueDB`.`tq_TaskQueues` (`TQId`);
ALTER TABLE `TaskQueueDB`.`tq_TQToJobTypes` ADD PRIMARY KEY (`TQId`, `Value`);
ALTER TABLE `TaskQueueDB`.`tq_TQToJobTypes` ADD FOREIGN KEY (`TQId`) REFERENCES `TaskQueueDB`.`tq_TaskQueues` (`TQId`);
ALTER TABLE `TaskQueueDB`.`tq_TQToTags` ADD PRIMARY KEY (`TQId`, `Value`);
ALTER TABLE `TaskQueueDB`.`tq_TQToTags` ADD FOREIGN KEY (`TQId`) REFERENCES `TaskQueueDB`.`tq_TaskQueues` (`TQId`);

Pilots Logging service:

The v6r20 release introduces a new OPTIONAL feature: the pilots logging service, which is PARTIALLY available only to users of so-called "Pilots 3". The pilots logging service is very well documented in the official DIRAC documentation, so please refer to that. Operationally, if you want to enable it, you will need an MQ system (like RabbitMQ, but not necessarily), and to do the following steps:

Define the new DB PilotsLoggingDB:

The new DB PilotsLoggingDB should be defined, then the tables in the databases should be created by the PilotsLogging service

Add new service:

new PilotsLogging service (via sysadmin-cli): install service WorkloadManagement PilotsLogging

RequestManagementSystem

DB changes

To accommodate the stricter MySQL 5.7:

use ReqDB;
ALTER TABLE Request MODIFY Error VARCHAR(2048);
ALTER TABLE Operation MODIFY Error VARCHAR(2048);
ALTER TABLE File MODIFY Error VARCHAR(2048);

Resources

Computing Elements

The Singularity Computing Element has been added within https://github.com/DIRACGrid/DIRAC/pull/3476

There's no specific changes that needs to be done server-wise.

TransformationSystem (v6r20p5, v6r20p7)

Please take note about the important changes to default values for the shifterProxy in some agents of the TransformationSystem https://github.com/DIRACGrid/DIRAC/wiki/DIRAC-v6r20p7

Add a custom footer
Add a custom sidebar
Clone this wiki locally