Add PVC as storage option for LM-Eval results

[ruivieira]

For large, full-results (not just summaries) storing the evaluation results in the CR's status is not practical.
An option should be available to persist full evaluation results to PVC.

[yhwang]

Here is my thought on this feature, suggestions/comments are welcome

• provide an option to save the outputs, including stdout, stderr, results, and sampling log, to external storage
• for external storage, we can support different types of storage, i.e. PVC, COS, etc. Let's add the PVC for now.

For the argument's sake, the new option is named Outputs and users can specify a full data struct of the PersistentVolumeClaim . LM-Eval controller would create the PVC and mount it on the /opt/app-root/src/output and all the outputs would store to the PVC. The data struct for the Outputs could be:

type Outputs struct {
	// +optional
	PersistentVolumeClaim *corev1.PersistentVolumeClaim `json:"pvc,omitempty"`
}

Only support PVC for now.

One thing worth mentioning is If users want to create a PVC by themselves and then use it in an LM-Eval job, they can do that already. Here is an example and the name of the PVC that has been created is mypvc:

apiVersion: trustyai.opendatahub.io/v1alpha1
kind: LMEvalJob
metadata:
  name: evaljob-sample
spec:
  model: hf
  modelArgs:
  - name: pretrained
    value: google/flan-t5-base
  taskList:
    taskRecipes:
    - card:
        name: "cards.wnli"
      template: "templates.classification.multi_class.relation.default"
  logSamples: true
  pod:
    volumes:
      - name: pvc
        persistentVolumeClaim:
          claimName: mypvc
    container:
      volumeMounts:
      - mountPath: "/opt/app-root/src/output"
        name: PVC

[yhwang]

If we want to make the case of using an existing PVC easier, the Outputs data struct could be:

type Outputs struct {
	// Use an existing PVC to store the outputs
	// +optional
	PersistentVolumeClaimName *string `json:"pvcName,omitempty"`
	// Create a PVC and use it to store the outputs
	// +optional
	PersistentVolumeClaim *corev1.PersistentVolumeClaim `json:"pvc,omitempty"`
}

When the pvcName is specified, the LM-Eval controller will create the corresponding volumes and volumeMOunts for the job pod and store the outputs in the PVC. Basically, it makes the LMEvalJob example in previous comment become this:

apiVersion: trustyai.opendatahub.io/v1alpha1
kind: LMEvalJob
metadata:
  name: evaljob-sample
spec:
  model: hf
  modelArgs:
  - name: pretrained
    value: google/flan-t5-base
  taskList:
    taskRecipes:
    - card:
        name: "cards.wnli"
      template: "templates.classification.multi_class.relation.default"
  logSamples: true
  outputs:
    pvcName: mypvc

[ruivieira]

Hi @yhwang, thanks for the suggestions. I think that's a good idea.
However, I've added a proposal with #347 that's slightly different, wanted to know your opinion.

IMO the two use cases (for the PVC scope, at the moment) are:

• PVC managed by the operator with minimal configuration
• Give user the ability to provide more specific PVC configurations

I think that adding the ability to specify a full PVC spec under the LMEvalJob CR might be avoided with your pvcName suggestion. This adds the freedom to create any custom PVC and just pass the name reference to the LMEvalJob.
For the scenario where we need a PVC to created from the LMEvalJob, we can simply use pvcManaged. wdyt?

[yhwang]

Your implementation looks great to me. Left a minor comment and I think it's good to go!