Large shared datasets

[vipasu]

Hi,

Does the data hub have storage options for large shared datasets? The use case is particle collider data or astrophysical datasets (possibly order ~terabytes). There are usually public available catalogs, but doesn't make sense to have every user have their own copy.

Thanks,
Vincent

[ryanlovett]

There are ways of accommodating shared datasets so that multiple users have read-only access to the same files. Budget is the limiting factor when it comes to the size. What class is this for? (year, term, course number)

[vipasu]

Don't have a specific course in mind for now, but there is lots of interest from the discussions with other physics professors. Likely will be physics 129 (particle physics) or physics 112 (statistical mechanics).

For budgeting, is it a one time semester fee or monthly? Does it scale with users?

Feel free to close this issue and I'll let the instructors know to file a new issue when they have a more specific request.

[yuvipanda]

Picking this back up! @aculich laid out possible naming conventions here.

To begin with, I'm setting up a shared directory for astro 128/256. The convention I'm using for this particular case is:

ReadOnly mount: shared/course/astr-128-256
ReadWrite mount: shared-rw/course/astr-128-256

The ReadOnly mount is available for everyone, while the ReadWrite mount is only available for 'admins' of a course. Since we don't have a concept of 'course admin', we will have to manually list them now.

In addition, a directory for this needs to be manually created on the NFS server for now. You can do this with:

mkdir -p /export/datahubhomes-2020-07-29/homes/_shared/course/astr-128-256
sudo chown 1000:1000 /export/datahubhomes-2020-07-29/homes/_shared/course/astr-128-256

[ryanlovett]

I wanna give a plug to install: 😄
install -d -o 1000 -g 1000 /export/datahubhomes-2020-07-29/homes/_shared/course/astr-128-256

[ryanlovett]

Closing, but please open a new issue when this is needed for a specific class.