Home

$${\color{red}\huge These \space pages \space are \space obsolete.\space We\space have\space migrated\space WMCore\space documentation\space to\space CERN\space server.}$$

New documentation URL: https://cms-wmcore.docs.cern.ch/

New repository URL: https://gitlab.cern.ch/dmwm/wmcore-docs

Introduction

This wiki is meant to provide a description as complete as possible of the different applications of the Workflow Management system (ReqMgr2, WMStats, GlobalWorkQueue, WMAgent) on what they do and how they do it. (WMCore library is also used for other applications, DBS,CRABServer CRABClient and WmAgentScript

If you are a developer, this should help you get on track to start making changes to the code but you must also checkout the developers tutorials for DMWM. Finally, there is no better documentation than the code itself if you want to work with the project, you'll likely spend many hours just going through it even after reading this.

If you are a non-developer, this will give you basic overview on WMAgent System.

General information for both users (data operators) and developers

Workflow Management System Overview

The following pages contain detailed information about the different projects and how they work.

• WM Overview
• WMAgent
• ReqMgr
• WorkQueue
• WMStats

MicroServices documentation

Microservices are also key elements in the Workload Management system. Documentation for the existent microservices follows.

• MicroService Transferor
• MicroService Monitor
• MicroService Output
• MicroService Unmerged
• MicroService RuleCleaner
• MicroService Pileup

Some important logics and concept on the system.

This section contains special pages which describe a particular phenomenon in the system, usually the information here can be inferred after reading the project pages completely.

• Workflow creation and assignment definition

• Request Types and Arguments

• TaskChain vs StepChain

• StepChain Parentage

• How is a request divided to jobs and submitted?

• How is the location of data resolved?

• Deterministic Pileup

• TimePerEvent convention

• Job priority handling in Agent system

• GPU Support in WMCore

Installation (Deployment):

Deploy agent (assume condor is already installed and connected to global pool. For Oracle backend, oracle account should already set up.

• WMAgent deployment script
• How to patch production machine
• DMWM deploy document

Usage and APIs

• Injecting, assigning and validating test requests
• Store result requests
• ReqMgr2 APIs
• WMStats APIs
• ReqMgr2 MicroService APIs

Operations-related documents

• various operation documents
• workqueue couch db replication procedure

Monitoring documentation

• WMAgent monitoring documentation
• Global WorkQueue monitoring documentation
• ReqMgr2 monitoring documentation
• Prometheus AlertManager wrapper APIs
• Monitoring of CMSWEB services with Prometheus AlertManager

Data Management related documentation

• Rucio rules created within the WM system
• Continuous input data deletion in Rucio

Trouble shooting

• WMAgent/ReqMgr/WorkQueue trouble shooting

Information for Developers

Development Tutorial

• New WMCore member setup
• WMCore developer responsibilities
• Setting up WMCore developing environment using github
• Useful git commands

Test setup

• How to set up unittest env (linux)
• All about Jenkins and our automated tests
• How to patch production/test machine (apply patches to WMAgent)
• Injecting, assigning and validating test requests
• Debugging, log locations
• Give access to other users to your vm

Issue tracking and release procedure

• Tagging and Releasing
• Creating issues

CMS Development Tools and Support

• DMWM tutorial
• DMWM build document
• DMWM deploy document

Some Implementation details

• WMBS (db backend for WMAgent)
• CouchDB Diagram
• Workqueue replication procedure
• JobSplitting Argorithms

On going developments

DMWM deployment transition (packaging with PyPi and using docker)

• WMCore in Kubernetes
• Improving WMAgent deployment (temporary transitional plan)
• DWWM packaging with PyPi
• WMCore with Docker

Python3 Transition

• list of things need to be done manually
• List of futurize stage2 fixes

PhEDEx to Rucio Transition

• PhEDEx to Rucio Transition
• Rucio data management plan

WMAgent Refactoring (dividing jobs in work units)

• Overview
• Implementation

Unified Porting

• Plan
• Transferor module

Go transition

• WorkQueue GO implementation

Miscellaneous

• Building a new python spec file with dependency
• Profiling the memory usage of a function

Deprecated and/or outdated documents

Just keeping them for the information and reference purposed or need to be updated and put in a proper category.

• How to set up all-in-one test
• BigCouch setup
• Draining Steps
• Deployment legacy instructions