-
Notifications
You must be signed in to change notification settings - Fork 107
Request Manager (ReqMgr2) REST APIs
This wiki provides some instructions and documentation on the Request Manager 2 RESTful APIs.
A general requirement of the WMCore REST framework is that clients must provide a Accept HTTP header in their request. Hence, if the end user wants to retrieve data in a JSON format, it needs to provide the following HTTP header request: Accept: application/json.
Starting in the WMCore release HG2211 - from November, 2022 - using WMCore version 2.1.4, the capability of gzip compressed response has been added to the WMCore REST framework. End users are invited to request compressed data, especially for heavy APIs transferring (many) megabytes of data, including most of the wmstatsserver RESTful APIs.
When the user is creating their HTTP request, an extra HTTP header has to be provided to communicate to the WMCore server that gzip'ed content is accepted by the client. The user has to provide this key/value parameter in their HTTP request: Accept-Encoding: gzip.
This does not necessarily mean that the server will, so the user must check the HTTP response headers to decide how to read the response body, which might or not be compressed. In case the server has responded with compressed data, the following HTTP response header will be sent back to the client Content-Encoding: gzip, flagging that that response body is in a binary/compressed format.
In order to decompress the body data, the client can use the gzip third-party python library and decompress the data as:
gzip.decompress(body)
If HTTP requests are made with the curl Unix tool, the same header has to be provided and the output data can be redirected to a file, example:
curl -L -k --cert $X509_USER_CERT --key $X509_USER_KEY --cacert $X509_USER_CERT https://cmsweb.cern.ch/reqmgr2/data/status -vvv -H "Accept: application/json" -H "Accept-Encoding: gzip" > out.data
now to see the content of out.data, one can use the zcat tool, example:
zcat out.data
To summarize the use of gzip, the client needs to provide the correct Accept-Encoding HTTP request header and when parsing the HTTP response object, a check for the HTTP response header Content-Encoding is required to decide how to deal with that object.
- To create the request - need POST request
POST /reqmgr2/data/request HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch
json format of data example: ReReco example
- To clone the request - need POST request
POST /reqmgr2/data/request/clone/[request_name] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch
i.e.
curl --cert $X509_USER_CERT --key $X509_USER_KEY -X POST -H "Content-type: application/json" -H "Accept: application/json" -d '{"Step1":{"RequestNumEvents": 4000}}' https://cmsweb.cern.ch/reqmgr2/data/request/clone/wmagent_dmason_NERSCTEST_171102_171002_8020
- update ([] here means user input not the list)
PUT /reqmgr2/data/request/[request_name] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch
To update request priority
i.e PUT /reqmgr2/data/request/[request_name]
Content-type: application/json
Host: cmsweb.cern.ch
Body: {"RequestPriority": 112000}
curl --cert $X509_USER_CERT --key $X509_USER_KEY -X PUT -H "Content-type: application/json" -d '{"RequestPriority": 112000}' https://cmsweb.cern.ch/reqmgr2/data/request/pdmvserv_task_PPD-RunIISpring18wmLHEGS-00001__v1_T_180220_111353_7996
json format of data example: ReReco example
4. To assign request with other properties
i.e PUT /reqmgr2/data/request/[request_name]
Content-type: application/json
Host: cmsweb.cern.ch
Body: {"RequestStatus": "assigned", "SiteWhitelist": ["T1_US_FNAL"], ...}}
curl --cert $X509_USER_CERT --key $X509_USER_KEY -X PUT -H "Content-type: application/json" -d "@assingment.json" https://cmsweb.cern.ch/reqmgr2/data/request/pdmvserv_task_PPD-RunIISpring18wmLHEGS-00001__v1_T_180220_111353_7996
GET /reqmgr2/data/request?name=[request_name] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch
GET /reqmgr2/data/request?status=[request_status] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch
* you can't retrieve *-archived status with this call. There is special status **ACTIVE** which retrieves all non archived status
GET /reqmgr2/data/request?status=[request_status]&request_type=[request_type] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch
* you can't just use request_type only it needs to combined with status
GET /reqmgr2/data/request?status=[request_status]&team=[team] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch
* you can't just use team only it needs to combined with status
** following calls can be combined with other key
GET /reqmgr2/data/request?prep_id=[prep_id] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch
GET /reqmgr2/data/request?inputdataset=[inputdataset] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch
GET /reqmgr2/data/request?outputdataset=[outputdataset] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch
GET /reqmgr2/data/request?campaign=[campaign] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch
GET /reqmgr2/data/request?mc_pileup=[mc_pileup] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch
GET /reqmgr2/data/request?data_pileup=[data_pileup] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch
* You can also filter out output by adding mask following will return only OutputDatasets and InputDatasets
GET /reqmgr2/data/request?data_pileup=[data_pileup]&mask=OutputDatasets&mask=InputDataset HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch
return format
{"result": [ {requestname: {key: value}, ....},
{requestname: {key: value}, ....}, ...]}
example)
{"result": [
{"pdmvserv_TSG-RunIISpring15MiniAODv2-00006_00082_v0__151012_002749_6757":
{"InputDataset": "/QCD_Pt-170to300_EMEnriched_TuneCUETP8M1_13TeV_pythia8/RunIISpring15DR74-Asympt25ns_MCRUN2_74_V9-v1/AODSIM",
"KeepStepOneOutput": true,
"Group": "ppd",
"CustodialSites": [],
"TimePerEvent": 1.5,
"Requestor": "pdmvserv",
"ProcessingString": "74X_mcRun2_asymptotic_v2",
"ScramArch": "slc6_amd64_gcc491",
"SizePerEvent": 60,
"Memory": 1980,
"PrepID": "TSG-RunIISpring15MiniAODv2-00006",
"AutoApproveSubscriptionSites": ["T1_US_FNAL_Disk"],
"CustodialSubType": "Replica",
"RequestType": "ReDigi",
"OpenRunningTimeout": 43200,
"InputDatasetTypes": {},
"OutputDatasets": ["/QCD_Pt-170to300_EMEnriched_TuneCUETP8M1_13TeV_pythia8/RunIISpring15MiniAODv2-74X_mcRun2_asymptotic_v2-v1/MINIAODSIM"],
"TotalInputEvents": 5735584,
"RequestDate": [2015, 10, 11, 22, 27, 49],
"AcquisitionEra": "RunIISpring15MiniAODv2",
"PrimaryDataset": "QCD_Pt-170to300_EMEnriched_TuneCUETP8M1_13TeV_pythia8",
"RequestPriority": 85000,
"SiteWhitelist": ["T1_IT_CNAF", "T1_FR_CCIN2P3", "T2_CH_CSCS", "T1_US_FNAL", "T1_UK_RAL"],
"SubscriptionPriority": "Low",
"ProcessingVersion": "1",
"Teams": ["production"],
"TotalEstimatedJobs": 308,
"RequestTransition": [{"Status": "new", "DN": null, "UpdateTime": 1444602475},
{"Status": "assignment-approved", "DN": null, "UpdateTime": 1444602478},
{"Status": "assigned", "DN": null, "UpdateTime": 1444626489},
{"Status": "acquired", "DN": null, "UpdateTime": 1444628716},
{"Status": "running-open", "DN": null, "UpdateTime": 1444634598}], i
"RequestName": "pdmvserv_TSG-RunIISpring15MiniAODv2-00006_00082_v0__151012_002749_6757",
...}
]}
GET /reqmgr2/data/wmagentconfig/[agent_host_name] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch
An example of a curl command to read the agent configuration is:
curl --cert $X509_USER_CERT --key $X509_USER_KEY -X GET https://alancc7-cloud1.cern.ch/reqmgr2/data/wmagentconfig/vocms0261.cern.ch
The complete content needs to be provided in the request body
POST /reqmgr2/data/wmagentconfig/[agent_host_name] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch
in request body, something like {'UserDrainMode':True, 'key2': 'value2', ...}
If the document already exists, the call will return a 'Document conflict' error and no changes will be made.
Example creating a new agent configuration:
curl --cert $X509_USER_CERT --key $X509_USER_KEY -X POST -H "Content-type: application/json" -d '{"ALAN":true, "MALTA":[1,2,3]}' https://alancc7-cloud1.cern.ch/reqmgr2/data/wmagentconfig/vocms0222.cern.ch
Following the good practices of REST implementation, the whole content needs to be provided in the request body. This PUT request replaces the old document by a new one with the new content.
PUT /reqmgr2/data/wmagentconfig/[agent_host_name] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch
in request body, something like {'UserDrainMode':True, 'key2': 'value2', ...}
For these auxiliary specific APIs, if the resource does not exist yet, a new document gets created.
Example for an agent document update:
curl --cert $X509_USER_CERT --key $X509_USER_KEY -X PUT -H "Content-type: application/json" -d '{"UserDrainMode":true, "ALAN":"test"}' https://alancc7-cloud1.cern.ch/reqmgr2/data/wmagentconfig/vocms0222.cern.ch
GET /reqmgr2/data/campaignconfig/[CAMPAIGN_NAME] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch
The complete content needs to be provided in the request body
POST /reqmgr2/data/campaignconfig/[CAMPAIGN_NAME] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch
in request body, something like {'CampaignName':'test', 'datasets': ['blah'], ...}
If the document already exists, the call will return a 'Document conflict' error and no changes will be made.
Note that the whole content needs to be provided in the request body. The PUT request replaces the old document by a new one with the new content.
PUT /reqmgr2/data/campaignconfig/[CAMPAIGN_NAME] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch
in request body, something like {'UserDrainMode':True, 'key2': 'value2', ...}
For these auxiliary specific APIs, if the resource does not exist yet, a new document gets created.
GET /reqmgr2/data/unifiedconfig/[UNIFIED_NAME] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch
The complete content needs to be provided in the request body
POST /reqmgr2/data/unifiedconfig/[UNIFIED_NAME] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch
in request body, something like {'goodsites':'test', 'key1': 'value1', ...}
If the document already exists, the call will return a 'Document conflict' error and no changes will be made.
Note that the whole content needs to be provided in the request body. The PUT request replaces the old document by a new one with the new content.
PUT /reqmgr2/data/unifiedconfig/[UNIFIED_NAME] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch
in request body, something like {'UserDrainMode':True, 'key2': 'value2', ...}
For these auxiliary specific APIs, if the resource does not exist yet, a new document gets created.
GET /reqmgr2/data/transferinfo/[TRANSFER_DOC_NAME] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch
If one wants to fetch all transfer documents in one single call, then the document name has to be ALL_DOCS, e.g.:
GET /reqmgr2/data/transferinfo/ALL_DOCS HTTP/1.1
The complete content needs to be provided in the request body
POST /reqmgr2/data/transferinfo/[TRANSFER_DOC_NAME] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch
in request body, something like {'workflowName':'test', ...}
If the document already exists, the call will return a 'Document conflict' error and no changes will be made.
Note that the whole content needs to be provided in the request body. The PUT request replaces the old document by a new one with the new content.
PUT /reqmgr2/data/transferinfo/[TRANSFER_DOC_NAME] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch
in request body, something like {'workflowName':'test', 'key2': 'value2', ...}
For these auxiliary specific APIs, if the resource does not exist yet, a new document gets created.
This API returns a list of parent datasets that are in use by workflows with "IncludeParents": True and the following statuses:
['assignment-approved', 'assigned', 'staging', 'staged', 'acquired', 'failed',
'running-open', 'running-closed', 'force-complete', 'completed', 'closed-out']
A list of InputDatasets for these workflows is obtained via a CouchDB view and DBS is queried via DBSReader.listDatasetParents() to create the list of parent datasets.
The REST API is parentlocks
GET /reqmgr2/data/parentlocks HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch
Example output
{"result": [
{
"ConfigType": "PARENT_LOCKS",
"parentlocks": [
"/Cosmics/Commissioning2015-v1/RAW",
"/HLTPhysics/Run2017B-v1/RAW"
]
}]}