[v6r20] MultiVO TransformationSystem #3723

andresailer · 2018-06-14T08:38:14Z

These changes make the WorkflowTaskAgent and RequestTask agent work without a shifterProxy by setting the owner as the delegatedDN/group for all the submission clients to the WMS and RMS.
This should be completely optional. If shifterProxy is set things should work as before.

TaskManagerAgentBase: add option ShifterCredentials to set the credentials to use for all submissions, this is single VO only
TaskManagerAgentBase: WorkflowTasks/RequestTasks: pass ownerDN and ownerGroup parameter to all the submission clients if using shifterProxy ownerDN and ownerGroup are None thus reproducing the original behaviour
TaskManagerAgentBase: refactor adding operations for transformation to separate function to ensure presence of Owner/DN/Group in dict entries
RequestTaskAgent no longer sets shifterProxy by default.

Plus some refactoring and minor fixes

Addresses #3695

coveralls · 2018-06-14T08:55:14Z

Coverage increased (+0.003%) to 20.607% when pulling dc8ddf9 on andresailer:multiVoSquashed into 6057929 on DIRACGrid:rel-v6r20.

chaen · 2018-06-14T13:08:51Z

TransformationSystem/Agent/TaskManagerAgentBase.py

@@ -110,6 +111,27 @@ def execute(self):
    """

    operationsOnTransformationDict = {}
+    owner, ownerGroup, ownerDN = None, None, None
+    # getting the credentials for submission
+    if getProxyInfo(False, False)['OK']:  # there is a shifterProxy


Can you name the parameters please ?

chaen · 2018-06-14T13:09:26Z

TransformationSystem/Agent/TaskManagerAgentBase.py

+    owner, ownerGroup, ownerDN = None, None, None
+    # getting the credentials for submission
+    if getProxyInfo(False, False)['OK']:  # there is a shifterProxy
+      res = getProxyInfo(False, False)


You could call that only once for the test and the result ?

chaen · 2018-06-14T13:14:57Z

TransformationSystem/Client/TaskManager.py


    if not requestClient:
-      self.requestClient = ReqClient()
+      self.requestClient = ReqClient(useCertificates=useCertificates,
+                                     delegatedDN=ownerDN,


Pfiou :-) Have you seen this actually used somewhere, or do you just trust the comments I've put a couple of years ago ? :D

In fact I am not sure how this works.
Do you set the ownerDN and the ownerGroup of the Request somewhere ? If not, it tries to get it from proxyInfo, which would not do anything now if I understand properly.
However, with the rpcStub fix that was merged lately, I am not sure of what will happen with these delegatedDN

Do you mean these comments? 🤣

DIRAC/Core/DISET/private/BaseClient.py

Lines 52 to 53 in 6926fc6

:param delegatedDN: Not clear what it can be used for.

:param delegatedGroup: Not clear what it can be used for.

I saw those, laughed, looked at git ~~blame~~ praise, tried it out and saw that it worked. Otherwise I would have given up on this route.

I also thought about the stub changes and rebased my developments yesterday, things are still working.

The ownerDN/group of the requests are (already) set in the RequestTasks TaskManager

DIRAC/TransformationSystem/Client/TaskManager.py

Lines 267 to 282 in 6926fc6

def _assignRequestToTask(self, oRequest, taskDict, transID, taskID, ownerDN, ownerGroup):

"""set ownerDN and group to request, and add the request to taskDict if it is

valid, otherwise remove the task from the taskDict

:param oRequest: Request

:param dict taskDict: dictionary of tasks, modified in this function

:param int transID: Transformation ID

:param int taskID: Task ID

:param str ownerDN: certificate DN used for the requests

:param str onwerGroup: dirac group used for the requests

:returns: None

"""

oRequest.RequestName = self._transTaskName(transID, taskID)

oRequest.OwnerDN = ownerDN

oRequest.OwnerGroup = ownerGroup

:D yes, these.
Good that it works. Not sure what you are praising though ! :D

fstagni · 2018-06-19T10:43:56Z

The looks OK to me, but you need to add info here: https://github.com/DIRACGrid/DIRAC/wiki/DIRAC-v6r21 and in the documentation, both the technical docu of the agents and in the /docs part.

andresailer · 2018-06-20T08:44:39Z

both the technical docu of the agents and in the /docs part.

What do you mean with technical doc ? configuration parameters etc?

andresailer · 2018-06-20T12:19:26Z

See here for doc example:
https://diracsailer.readthedocs.io/en/multivosquashed/CodeDocumentation/TransformationSystem/Agent/WorkflowTaskAgent.html

andresailer · 2018-06-20T12:20:29Z

For proper working of the RTA we need a fix for #3727

fstagni · 2018-06-20T15:45:08Z

Yes I meant what you have done: config parameters inside the code.

andresailer · 2018-06-21T12:08:47Z

Added the config like parameter descriptions for direct comparison, will remove the other one later on.
(moving from table to this is more effort than manipulating the table in emacs)

https://diracsailer.readthedocs.io/en/multivosquashed/CodeDocumentation/TransformationSystem/Agent/RequestTaskAgent.html (clear your cache if you don't see it)

Also realised that the doc was wrong two options are actually UpdateFilesStatus and UpdateTasksStatus...

fstagni · 2018-06-21T12:04:34Z

TransformationSystem/Agent/TaskManagerAgentBase.py

@@ -254,12 +247,28 @@ def _getClients(self):
    return {'TransformationClient': threadTransformationClient,
            'TaskManager': threadTaskManager}

+  def _getDelegatedClients(self, ownerDN=None, ownerGroup=None):


Can you simply modify _getClients() instead of adding one more function? For extensions is one less thing to extend.

yes, moved the parameters into _getClients

andresailer · 2018-06-21T12:48:27Z

Also added to twiki https://github.com/DIRACGrid/DIRAC/wiki/DIRAC-v6r21#multivo-support-for-workflowtaskagent-and-requesttaskagent

andresailer · 2018-06-22T06:35:27Z

@atsareg any comment for the layout of the parameters? table vs config like structure?

fstagni · 2018-07-05T09:58:39Z

Will be rebased on v6r20

…cept operation is used in the except block

TaskManagerAgentBase: add option ShifterCredentials to set the credentials to use for all submissions, this is single VO only TaskManagerAgentBase: WorkflowTasks/RequestTasks: pass ownerDN and ownerGroup parameter to all the submission clients if using shifterProxy ownerDN and ownerGroup are None thus reproducing the original behaviour TaskManagerAgentBase: refactor adding operations for transformation to separate function to ensure presence of Owner/DN/Group in dict entries

execute is running after _execute(?), so we need to find ShifterCredentials in initialize

…file

…g template like structure

andresailer · 2018-07-05T11:04:22Z

All done.
Also: https://github.com/DIRACGrid/DIRAC/wiki/DIRAC-v6r20#transformationsystem-v6r20p5

andresailer requested review from atsareg and fstagni as code owners June 14, 2018 08:38

andresailer force-pushed the multiVoSquashed branch from d416f69 to eb3473b Compare June 14, 2018 08:39

chaen reviewed Jun 14, 2018

View reviewed changes

andresailer force-pushed the multiVoSquashed branch 2 times, most recently from fbcc138 to c3655cc Compare June 18, 2018 15:07

andresailer mentioned this pull request Jun 20, 2018

[v6r20] Docs: Transformation: typos, added few phrases, bulk submission #3725

Merged

andresailer force-pushed the multiVoSquashed branch from c3ae97c to 47edd83 Compare June 20, 2018 12:17

andresailer force-pushed the multiVoSquashed branch from 47edd83 to a9934cf Compare June 20, 2018 12:27

andresailer force-pushed the multiVoSquashed branch 2 times, most recently from 9fd701c to e278278 Compare June 21, 2018 12:06

fstagni reviewed Jun 21, 2018

View reviewed changes

andresailer changed the title ~~[integration] MultiVO TransformationSystem~~ [v6r21] MultiVO TransformationSystem Jun 25, 2018

andresailer force-pushed the multiVoSquashed branch from 1fbef5f to 46f1769 Compare June 25, 2018 08:51

fstagni approved these changes Jul 3, 2018

View reviewed changes

fstagni changed the title ~~[v6r21] MultiVO TransformationSystem~~ [WIP][v6r21] MultiVO TransformationSystem Jul 5, 2018

andresailer changed the base branch from integration to rel-v6r20 July 5, 2018 10:50

andresailer force-pushed the multiVoSquashed branch from 43ebb78 to 9cb36a9 Compare July 5, 2018 10:50

andresailer requested a review from zmathe as a code owner July 5, 2018 10:50

TransferClient: remove weird ^M

d0a32a7

andresailer added 11 commits July 5, 2018 12:53

TaskManagerAgentBase: have to define operation variable before try/ex…

0e415b3

…cept operation is used in the except block

RequestTaskAgent: drop default value for shifterProxy

54393ab

TaskManagerAgentBase: name getProxyInfo Parameters; store result

4e9a652

TaskManagerAgents: fix logic for non-proxy based general credentials

332fb05

execute is running after _execute(?), so we need to find ShifterCredentials in initialize

Docs: TransformationSystem add MultiVO setup options

b4e8f23

WTA: Docs: move configuration parameter table to python file

143aebd

RequestTaskAgent: Docs: move configuration parameter table to python …

1ace00f

…file

RequestTaskAgent and WorkflowTaskAgent: Docs: add parameters as confi…

7aa0d35

…g template like structure

TaskManagerAgentBase: add ownerDN, ownerGroup parameters to _getClients

3859ad3

RequestTaskAgent and WorkflowTaskAgent: Docs: drop tables of parameters

830d860

andresailer force-pushed the multiVoSquashed branch from 9cb36a9 to 830d860 Compare July 5, 2018 10:53

TransformationDocs: downgrade multiVO versionadded to v6r20p5

dc8ddf9

andresailer changed the title ~~[WIP][v6r21] MultiVO TransformationSystem~~ [v6r20] MultiVO TransformationSystem Jul 5, 2018

atsareg approved these changes Jul 6, 2018

View reviewed changes

atsareg merged commit b8d9fc7 into DIRACGrid:rel-v6r20 Jul 6, 2018

andresailer deleted the multiVoSquashed branch July 6, 2018 15:32

fstagni mentioned this pull request Nov 2, 2019

[v7r0] RSS command fixes #4336

Merged

chaen mentioned this pull request Feb 9, 2023

[8.1] feat: RequestManagementSystem: remove DN in favour of owner #6653

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[v6r20] MultiVO TransformationSystem #3723

[v6r20] MultiVO TransformationSystem #3723

andresailer commented Jun 14, 2018 •

edited

Loading

coveralls commented Jun 14, 2018 •

edited

Loading

chaen Jun 14, 2018

chaen Jun 14, 2018

chaen Jun 14, 2018

chaen Jun 14, 2018

andresailer Jun 14, 2018

chaen Jun 14, 2018

fstagni commented Jun 19, 2018

andresailer commented Jun 20, 2018 •

edited

Loading

andresailer commented Jun 20, 2018

andresailer commented Jun 20, 2018

fstagni commented Jun 20, 2018

andresailer commented Jun 21, 2018

fstagni Jun 21, 2018

andresailer Jun 21, 2018

andresailer commented Jun 21, 2018

andresailer commented Jun 22, 2018

fstagni commented Jul 5, 2018

andresailer commented Jul 5, 2018

	:param delegatedDN: Not clear what it can be used for.
	:param delegatedGroup: Not clear what it can be used for.

	def _assignRequestToTask(self, oRequest, taskDict, transID, taskID, ownerDN, ownerGroup):
	"""set ownerDN and group to request, and add the request to taskDict if it is
	valid, otherwise remove the task from the taskDict

	:param oRequest: Request
	:param dict taskDict: dictionary of tasks, modified in this function
	:param int transID: Transformation ID
	:param int taskID: Task ID
	:param str ownerDN: certificate DN used for the requests
	:param str onwerGroup: dirac group used for the requests
	:returns: None
	"""

	oRequest.RequestName = self._transTaskName(transID, taskID)
	oRequest.OwnerDN = ownerDN
	oRequest.OwnerGroup = ownerGroup

[v6r20] MultiVO TransformationSystem #3723

[v6r20] MultiVO TransformationSystem #3723

Conversation

andresailer commented Jun 14, 2018 • edited Loading

coveralls commented Jun 14, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fstagni commented Jun 19, 2018

andresailer commented Jun 20, 2018 • edited Loading

andresailer commented Jun 20, 2018

andresailer commented Jun 20, 2018

fstagni commented Jun 20, 2018

andresailer commented Jun 21, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andresailer commented Jun 21, 2018

andresailer commented Jun 22, 2018

fstagni commented Jul 5, 2018

andresailer commented Jul 5, 2018

andresailer commented Jun 14, 2018 •

edited

Loading

coveralls commented Jun 14, 2018 •

edited

Loading

andresailer commented Jun 20, 2018 •

edited

Loading