Project: MOC Proof-of-Concept

[cmbz]

Purpose

• Deploy containerized Dataverse on Mass Open Cloud to support large data and computing
• Present demo of proof-of-concept at the Mass Open Cloud Alliance Conference on 2024/02/28

Background

• Mass Open Cloud Alliance is a "partnership between higher education, medical research centers, government, and industry provides structure and resources that enable a close collaboration between research, development, and operations in a series of interconnected projects."
• The MOC team wants to test a containerized Dataverse installation on the MOC infrastructure to support large data and computing.

Participants

• Leonid Andreev, @landreev
• Gustavo Durand, @scolapasta
• Sonia Barbosa, @sbarbosadataverse
• Ceilyn Boyd, @cmbz
• Stefano Iacus, @siacus
• Max Shad
• James Culbert
• Milson Munakami
• Jim Myers, @qqmyers
• Milan Kupcevic

Timeline

• Start: 2024/01/08
  • Milestone: Deliver containerized Dataverse installation (2024/01/15)
  • Milestone: Binder equivalent functionality implemented (2024/01/31)
  • Upload presentation (2024/02/23)
  • Milestone: Presentation (2024/02/24)
  • Close out tasks
• End: 2024/04/01

Tasks

• Create NERC project (@siacus)
• Project management: @cmbz and @scolapasta and @siacus coordinate re. resource allocation
  • Assemble team
  • GEOS-CHEM use case (see Project: GEOS-Chem Proof-of-Concept #154)
  • Set up recurring meetings, if needed (Slack channel used instead: https://iqss.slack.com/archives/C06EXB41EE5)
• Set up Dataverse container/installation for MOC (DevOps, @landreev)
• Arrange for authentication support for Keycloak
• Move from Binder to local NERC infrastructure to run workflows (Development, @pdurbin)
• Arrange for support of NESE tape connection
• Finalize presentation (@siacus)

Issues

• MOC PoC: Set up a VM for Dataverse entity for the demo #172

Related

• Epic: GREI 5 - Use Cases #119
• Proposal: Large Data Support #106
• Project: GEOS-Chem Proof-of-Concept #154
• Tutorial for running containers as a demo or for evaluation dataverse#10238
• (Experimental feature) Expose the location/path of the physical file via the Access api dataverse#10302

Technical Resources

• NESE tape resources
• NESE disk resources

Project Resources

• Google Drive Folder
• Running Meeting Notes
• Slack Channel
• Presentation Slides

[cmbz]

2024/01/29: Updated with new task: #172

[landreev]

A possible dependency: IQSS/dataverse#10302.
I can make a PR very quickly if/when needed.

[landreev]

Just a quick status checklist on the NERC deployment side:

• Basic Dataverse instance created (a Docker deployment in an OpenStack VM)
• Container Store configured (2TB)
• Access to the s3 bucket from within or outside the cluster
  • specifically, Dataverse instance configured w/ s3 access credentials
  • upload into the bucket via Dataverse tested
    • DvUploader works like a charm (tested with the GEOS-Chem data files used by the presentation notebook; see below)
    • uploads via the UI need to be re-tested
• The Computation node setup remains the major component that's still missing, so this is the main focus of the current effort
  • We got the Python notebook from GEOS-Chem. It's nowhere near ready to be deployable in a standalone OpenShift Jupyter pod as has been the plan so far. I will be working with the author. Project: GEOS-Chem Proof-of-Concept #154 will be used to track the effort.
  • Once we got the notebook deployment figured out, the "external tool" mechanism will need to be added to it to perform the basic initial interaction with the Dataverse for obtaining the storage locations of the files.
• NESE tape storage volume needs to be set up and configured on the Dataverse instance via Globus, as secondary storage for the demo
  • Created a dedicated Globus ID and had roles set up by NESE giving it access to the separate tape and disk pools, so I can configure the demo instance with these Globus credentials to facilitate data uploads to tape

[landreev]

edited the checklist to add the NESE tape volume.

[landreev]

Marked the "NESE tape" item as complete on the checklist - under the assumption that it is complete "for the purposes of the MOC PoC presentation slides". I'm opening a new issue dedicated to figuring out a working setup, with a functioning Borealis app hosted on the server, etc., that can be used by real users, that we can later replicate in IQSS prod.

[cmbz]

2024/03/01

• Closing issue as complete
• See follow up: Epic: Operationalize Large Data and Compute Infrastructure #184