demo: using grid proxy

[tiborsimko]

Provide complete REANA recipe for a demo workflow that needs to use grid proxy.

For example, calling dasgoclient for restricted CMS datasets.

(For CMS open datasets, cernopendata-client could in theory be used.)

See https://gitlab.cern.ch/clange/grid-proxy-test by @clelange

[tiborsimko]

Note: this is only for data access, not for computations. @alintulu might be looking into this.

[clelange]

From CMS Computing it seems that a better solution than everyone adding their certificates individually would be that REANA provides a proxy for all CMS users (checking egroup).

[roksys]

• Add proxy initialization logic to the reana-job-controller. See the Kerberos example https://github.com/reanahub/reana-job-controller/blob/ad7d17b4fc095f0cb53e925d568d4cb63078bfb4/reana_job_controller/kubernetes_job_manager.py#L173
  https://github.com/reanahub/reana-job-controller/blob/ad7d17b4fc095f0cb53e925d568d4cb63078bfb4/reana_job_controller/kubernetes_job_manager.py#L263
• Add proxy field to the serial schema (reana_commons/serial.py) and (reana_job_controller/schemas.py), generate new openapi_specifications/reana_job_controller.json by executing cd ../reana-job-controller && FLASK_APP=reana_job_controller/app.py flask openapi create openapi.json -p
• Add proxy filed to the reana-workflow-engine-[serial,yadage,cwl]. See the Serial example reanahub/reana-workflow-engine-serial@8200cf6#diff-6d29ceb19895382845929513c9bc9099
• Add docs - http://docs.reana.io/advanced-usage/access-control/kerberos

[lukasheinrich]

this is a similar discussion to the secrets / private images. Maybe a good option is that each reana user has access to

• their own personal secrets
• a common set of secrets managed by the deployment (perhaps matched by e-group)

[tiborsimko]

Yes, the user jobs have access to secrets, but then every user would have to run kinit and voms proxy etc friends themselves... This does not seem efficient and may be perhaps too low-level for some users?

The advantage of doing this automatically for the user via sidecar init containers is that the REANA system automatically takes care of setting everything up, including reauthentication for longer-living jobs. The user simply declares "I need Kerberos" via kerberos: true workflow clause, sets up his/her secret, and he/she does not have to care further and can focus on physics. (Similarly for grid proxy files.)

Philosophically speaking, this is perhaps another example of the declarative approach (I want kerberos: true and don't care how it is done) vs imperative approach (I want to run kinit and eventually k5reauth after 24 hours) in people's workflows?

[lukasheinrich]

yes I'm all for a declarative approach and I agree

* mount secret
* kinit

is less declarative then

* kerberos: true

or

* registry-access: atlas@gitlab-registry.cern.ch

or similar. I guess there is a little bit of a danger of introducing abstractions that only REANA@CERN can satisfy, but we can test this separately.