fix: clean up UTs for file metrics collector

[Electronic-Waste]

What this PR does / why we need it:

Current error comparison in file-metricscollector_test.go is based on a boolean rather than the error content To make test cases more detailed and accurate, we should expose and compare errors.

And also @tenzen-y suggested that t.TempDir should be used for generating test directory. So I replace static directories with t.TempDir.

Related discussion: #1756 (comment) #2274 (comment)

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #1756

Checklist:

• Docs included if any changes are user facing

[Electronic-Waste]

PTAL👀 /assign @andreyvelich @tenzen-y . BTW my flag is standing :)

[tenzen-y]

Give my first review

[Electronic-Waste]

I reordered the judging logics in CollectObservatoinLog, for we should check the fileFormat first in case of opening an invalid file. (Or the test case "Invalid file format" will report "can't find file or directory error` error since the order of test file generation is not sure.)

Also, some typical error's descriptions are added to the returned error in file-metricscollector.

PTAL👀 if you have some spare time @tenzen-y @andreyvelich .

[tenzen-y]

Basically, lgtm

[Electronic-Waste]

@tenzen-y It seems that the CI failed due to other reasons which are not concerned with my code.

[tenzen-y]

@tenzen-y It seems that the CI failed due to other reasons which are not concerned with my code.

I'll try to check it.

[tenzen-y]

@tenzen-y It seems that the CI failed due to other reasons which are not concerned with my code.

I'll try to check it.

Crated: #2297

[tenzen-y]

@Electronic-Waste Could you rebase this PR? Because I fixed CI issues.

[Electronic-Waste]

@tenzen-y Sure! I'll rebase it soon.

[Electronic-Waste]

Some unexpected errors occurred in Go Test / Unit Test (1.27.1) (pull_request)

{"level":"info","ts":"2024-04-03T01:41:34Z","logger":"trial-controller","msg":"Trial status changed to Metrics Unavailable","Trial":{"name":"test-trial","namespace":"default"}}
--- FAIL: TestReconcileBatchJob (82.16s)
    trial_controller_test.go:274: 
        Timed out after 80.000s.
        Expected
            <bool>: false
        to be true
FAIL

And also E2E Test with pytorch-mnist / e2e (v1.25.12, file-metrics-collector,pytorchjob-mnist):

NAME                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
katib-controller    ClusterIP   10.103.178.5     <none>        443/TCP,8080/TCP,18080/TCP   2s
katib-db-manager    ClusterIP   10.107.4.68      <none>        6789/TCP                     2s
katib-mysql         ClusterIP   10.103.235.10    <none>        3306/TCP                     2s
training-operator   ClusterIP   10.104.220.108   <none>        8080/TCP                     3s
Error from server (InternalError): error when creating "../../testdata/valid-experiment.yaml": Internal error occurred: failed calling webhook "defaulter.experiment.katib.kubeflow.org": failed to call webhook: Post "[https://katib-controller.kubeflow.svc:443/mutate-experiment?timeout=10s](https://katib-controller.kubeflow.svc/mutate-experiment?timeout=10s)": dial tcp 10.103.178.5:443: connect: connection refused

I encountered them many times in CI. Should we look into these errors and fix them?

[tenzen-y]

I encountered them many times in CI. Should we look into these errors and fix them?

Those errors are known flakness issues. I'll try to restart jobs.

[tenzen-y]

Thank you!
/lgtm
/approve

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Electronic-Waste, tenzen-y

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [tenzen-y]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Electronic-Waste]

@tenzen-y Thank you for your review!