Add 'triaged-jobs' to Pangeo Workspace

[anilnatha]

Is your feature request related to a problem? Please describe.

User's are unable to access triaged jobs in the Pangeo workspace

From notes doc:

Pangeo workspace missing triaged-jobs mount/s3fs lib

Describe the solution you'd like

User's should be able to access triaged jobs from the Pangeo workspace as is possible in other ADE workspaces.

Describe alternatives you've considered

None, yet.

Additional context

None, yet.

[anilnatha]

@wildintellect Please add any additional details about this ticket, if you have any.

[wildintellect]

Other workspaces like vanilla automatically have triaged-jobs mounted so users can access logs from failed jobs. This is really useful when debugging DPS fails. It makes sense to add this to most if not all ADE workspaces for consistency.
@anilnatha can you also check that this is a read-only mount with s3fs?

Also @rtapella this is completely undocumented.

[anilnatha]

Thank you for this additional information and I agree with your statement about having this available in most or all workspaces for consistency.

This functionality is new territory for me so may take me a little bit to become familiar with how it works.

[wildintellect]

I believe it's an s3fs mount similar to how shared-buckets work.

[anilnatha]

This appears to be a duplicate effort that @grallewellyn is already working on here: #656

[grallewellyn]

The tickets are loosely related, I was just testing in vanilla and didn't realize pangeo was missing triaged-jobs

This error is likely happening because the pangeo devfile is missing these lines

                  - name: s3fs-volume
                    mountPath: /projects/triaged-jobs
                    subPath: triaged-jobs
                    mountPropagation: HostToContainer

and

                  - name: s3fs-volume
                    mountPath: /triaged-jobs
                    subPath: triaged-jobs
                    mountPropagation: Bidirectional

This is a change that @bsatoriu knows how to make and deploy, and we can deploy the fix to pangeo when we address a fix needed for the other ticket that changes those lines to be mountPath: /projects/triaged_job since that is what makes the mounting consistent with the file path of the failed jobs (ie failed job gives "s3://.../triaged_job/dps_tutorial_graceal/main/2024/01/18/...")
I think we should keep this ticket open as a reminder that pangeo is missing triaged_job
@bsatoriu When you get a chance, can you please deploy this change? That is, that pangeo now has triaged_job and everything in the triaged-jobs folder is mounted as "triaged_job" in the user's ADE workspace

[grallewellyn]

I think we can move this to resolved? @bsatoriu