Skip to content

WorkQueue

Jump to bottom
ticoann edited this page May 9, 2018 · 3 revisions

The WorkQueue is a multi-level queue which represents work as CouchDB elements, better known as WorkQueueElements. These elements have key information about the work they represent such as input data, statistics of the amount of work it represents measured in number of jobs, a state and other operational data.

Any WMCore WorkQueue will consist of two databases in a CouchDB backend to store the work elements:

  • Inbox: The inbox represents the elements to be processed by the queue and that come from a higher entity, this can be either a parent queue or a provider of work such as the ReqMgr.
  • Queue: The database where processed inbox elements are moved to when they are ready for a lower level queue or processing by a WMAgent/WMBS.

REF : https://twiki.cern.ch/twiki/bin/view/CMS/WMCoreWorkQueue

Global WorkQueue

  1. A global WorkQueue takes work (requests) from a ReqMgr2 and
  2. Splits them in small pieces WorkQueue Elements (block) in order to make it available for a local queue to acquire and posteriorly inject it into WMBS.
  3. Provide estimated jobs fore each WQE which used for matching the resources.
  4. Updates locations of the WorkQueue Elements periodically
  5. Delete the WQEs when request is completed

The process starts by querying the configured ReqMgr instance for all the requests in assigned state for each team configured in the ReqMgr. For each request, two documents are created. One is a minimal WorkQueueElement in the global inbox that contains the follow information:

  • RequestName: Name of the request
  • Status: Status of the element, the initial state is Negotiating.
  • StartPolicy: Name and parameters of the splitting policy

The second is a CouchDB document in the global queue that contains the pickled spec for the request. This spec is a copy of the version that was stored in the ReqMgr database after assignment.

After creating these documents, the WorkQueue splits the Negotiating elements in the global queue according to the splitter configured in the StartPolicy. A splitter's task is to take a spec and create WorkQueueElements which hold a reasonable amount of work, for example 250 jobs in the MC case. The splitters are:

  • Block: A block splitter is used in ReReco, ReDigi, TaskChain with input dataset and MonteCarloFromGEN requests. A block splitter will take the input dataset in the spec, filter it through the block/run black/white lists and create a WorkQueueElement for each resulting closed block which has valid files in it. The amount of work in each element is determined depending on the job splitting which can be based on events, lumis or files and the resulting number of jobs is stored in the WorkQueueElement document for monitoring purposes. Initially the block locations are retrieved from DBS2.
  • MonteCarlo: A MonteCarlo splitter is used in MonteCarlo and LHEStepZero requests. The MonteCarlo splitter takes the requested number of events and divides it by the configured events per job to create a WorkQueueElement for each 250 jobs. The splitter also calculates the number of lumis that will be generated in each work element and stores the lumi ranges in the elements. Note that MonteCarlo WorkQueueElements have no input data.
  • ResubmitBlock: The ResubmitBlock splitter is used for resubmission requests, the splitter creates a single work element for all the ACDC records in the server for the given request and task from the resubmission request. This WorkQueueElement contains a reference to the ACDC documents to be used as input.

After the splitter is run over the inbox element, the resulting elements are created in the global queue with Available status. At this point, the inbox element is moved to Acquired status.

At this point the global WorkQueue is done with these elements and it expects for a local queue to acquire them and continue the process.

Continuous splitting

For the requests that use the Block splitter policy, it is possible for new input data to appear after the first split, e.g. open blocks that are closed or completely new blocks that appear in the input dataset. If the request is configured with an OpenRunningTimeout then the global WorkQueue will continue to examine the inbox elements for these requests and create new WorkQueueElements if new blocks appear. The splitting process is the same as described before, except that blocks already processed are ignored.

Additionally the WorkQueue scans the inbox to close requests that have been open for too longr. The criteria to mark a request as closed, i.e. move it from running-open to running closed is:

  • There are no valid (i.e. not blacklisted/in the whitelist) open blocks present in DBS.
  • The time since the last closed block was processed is more than the configured OpenRunningTimeout.

ReqMgr interaction

The WorkQueue must keep its state synced with the ReqMgr for successful operation. The WorkQueue takes care of moving the state of requests depending on changes in the WorkQueueElements or changing the WorkQueueElements status depending on changes in the state of a request. Here are the usual changes:

  • When an inbox element is split and the WorkQueueElements created in the global queue, then a request is moved from assigned to acquired.
  • When at least one of the elements of the global queue is injected into WMBS in a WMAgent (this means that the element has been replicated to a local queue, acquired and injected, and its status has been replicated back to the global queue) then the inbox element is marked as Running and the request is moved from acquired to running-open.
  • When a request is in running-open and its splitter is not Block then the request is moved to running-closed.
  • When a request is in running-open and the criteria for closing is met then the request is moved to running-closed.
  • When all the WorkQueueElements in a request are marked as Done, then the inbox element is marked as Done too and the request is moved from running-closed to completed.
  • When at least one WorkQueueElement is marked as failed then the inbox element is marked as failed an the request is too marked as failed.
  • When a request is marked as aborted, see the aborted article.

The WorkQueue also does cleanup operations on requests that are forcefully moved to states such as completed, it makes sure that elements from requests in terminal states are cleaned up when they reach terminal WorkQueueElement states such as Failed or Done.

Local WorkQueue

The local queues poll the global queue for elements according to configured teams and inject them into WMBS, this is better explained in the WMAgent section. Note that the local queue inbox replicates from the global queue so when an element changes status in a local inbox this information is propagated to the global queue and allows the WorkQueue to determine changes in the requests progress.

Clone this wiki locally