Unsupported Scan Error While Listing the Jobs of an Experiment

[vicaire]

Here is the raw HTTP:

GET /api/v1/namespaces/kubeflow/services/ml-pipeline:8888/proxy/apis/v1beta1/jobs?page_size=10&resource_reference_key.id=9333ecee-28b2-4c53-807d-bbfd2a45423f&resource_reference_key.type=EXPERIMENT HTTP/1.1
Host: 35.224.113.48
User-Agent: Go-http-client/1.1
Accept: application/json
Accept-Encoding: gzip

HTTP/2.0 500 Internal Server Error
Connection: close
Audit-Id: a32bebb7-3520-4819-9f2f-1003a0d39977
Content-Type: application/json
Date: Fri, 09 Nov 2018 08:55:32 GMT

{"error":"Failed to list jobs.: List jobs failed.: List data model failed.: InternalServerError: Failed to list jobs: sql: Scan error on column index 0, name "UUID": unsupported Scan, storing driver.Value type \u003cnil\u003e into type *string: sql: Scan error on column index 0, name "UUID": unsupported Scan, storing driver.Value type \u003cnil\u003e into type *string","code":13,"details":[{"@type":"type.googleapis.com/api.Error","error_message":"Internal Server Error","error_details":"Failed to list jobs.: List jobs failed.: List data model failed.: InternalServerError: Failed to list jobs: sql: Scan error on column index 0, name "UUID": unsupported Scan, storing driver.Value type \u003cnil\u003e into type *string: sql: Scan error on column index 0, name "UUID": unsupported Scan, storing driver.Value type \u003cnil\u003e into type *string"}]}
Raw error from the service: Failed to list jobs.: List jobs failed.: List data model failed.: InternalServerError: Failed to list jobs: sql: Scan error on column index 0, name "UUID": unsupported Scan, storing driver.Value type into type *string: sql: Scan error on column index 0, name "UUID": unsupported Scan, storing driver.Value type into type *string (code: 13)

[vicaire]

Checking that the experiment is valid:

go run main.go experiment list

• created_at: "0001-01-01T00:00:00.000Z"
  description: No description provided
  id: 9333ecee-28b2-4c53-807d-bbfd2a45423f
  name: default
• created_at: "0001-01-01T00:00:00.000Z"
  description: No description provided
  id: 04c888f6-a59d-4c94-ad24-e9555ee222c9
  name: totox
• created_at: "0001-01-01T00:00:00.000Z"
  description: No description provided
  id: bed43f9a-fd00-4f99-a0b8-269e0941ae56
  name: hello

[vicaire]

Additionally, it does not seem possible to list all the jobs:

GET /api/v1/namespaces/kubeflow/services/ml-pipeline:8888/proxy/apis/v1beta1/jobs?page_size=10&resource_reference_key.type=UNKNOWN_RESOURCE_TYPE HTTP/1.1
Host: 35.224.113.48
User-Agent: Go-http-client/1.1
Accept: application/json
Accept-Encoding: gzip

HTTP/2.0 400 Bad Request
Connection: close
Audit-Id: a0f5822d-f727-4d96-a6f2-fe6bbc19d788
Content-Type: application/json
Date: Fri, 09 Nov 2018 08:58:54 GMT

{"error":"Validating filter failed.: Unrecognized resource reference type.: Invalid input error: Unsupported resource type: UNKNOWN_RESOURCE_TYPE","code":3,"details":[{"@type":"type.googleapis.com/api.Error","error_message":"Unsupported resource type: UNKNOWN_RESOURCE_TYPE","error_details":"Validating filter failed.: Unrecognized resource reference type.: Invalid input error: Unsupported resource type: UNKNOWN_RESOURCE_TYPE"}]}

I think we should support that case.

[vicaire]

Unexpectedly, specifying an EXPERIMENT ID as the KEY ID and "JOB" as the KEY TYPE works (but does not return anything):

GET /api/v1/namespaces/kubeflow/services/ml-pipeline:8888/proxy/apis/v1beta1/jobs?page_size=10&resource_reference_key.id=9333ecee-28b2-4c53-807d-bbfd2a45423f&resource_reference_key.type=JOB HTTP/1.1
Host: 35.224.113.48
User-Agent: Go-http-client/1.1
Accept: application/json
Accept-Encoding: gzip

HTTP/2.0 200 OK
Content-Length: 2
Audit-Id: 27eb7f32-173e-45a7-93cc-0a645e777fd8
Content-Type: application/json
Date: Fri, 09 Nov 2018 09:00:12 GMT
Grpc-Metadata-Content-Type: application/grpc

{}

[vicaire]

If the experiment ID has no jobs, specifying an EXPERIMENT ID as the KEY ID and "EXPERIMENT" as the KEY TYPE seems to work:

GET /api/v1/namespaces/kubeflow/services/ml-pipeline:8888/proxy/apis/v1beta1/jobs?page_size=10&resource_reference_key.id=04c888f6-a59d-4c94-ad24-e9555ee222c9&resource_reference_key.type=EXPERIMENT HTTP/1.1
Host: 35.224.113.48
User-Agent: Go-http-client/1.1
Accept: application/json
Accept-Encoding: gzip

HTTP/2.0 200 OK
Content-Length: 2
Audit-Id: c184933a-fc1b-46ad-b3fc-ae739d6b835f
Content-Type: application/json
Date: Fri, 09 Nov 2018 09:03:20 GMT
Grpc-Metadata-Content-Type: application/grpc

{}
[]

[vicaire]

Trying again:

Creating a job:

go run main.go job create --pipeline-id 2dfffef5-dc47-4ae7-a0f4-1d23507f74c6 --name my-job --experiment-id bed43f9a-fd00-4f99-a0b8-269e0941ae56 --debug

POST /api/v1/namespaces/kubeflow/services/ml-pipeline:8888/proxy/apis/v1beta1/jobs HTTP/1.1
Host: 35.224.113.48
User-Agent: Go-http-client/1.1
Content-Length: 383
Accept: application/json
Content-Type: application/json
Accept-Encoding: gzip

{"created_at":"0001-01-01T00:00:00.000Z","description":"No description provided","enabled":true,"max_concurrency":"1","name":"my-job","pipeline_spec":{"parameters":[],"pipeline_id":"2dfffef5-dc47-4ae7-a0f4-1d23507f74c6"},"resource_references":[{"key":{"id":"bed43f9a-fd00-4f99-a0b8-269e0941ae56","type":"EXPERIMENT"},"relationship":"OWNER"}],"updated_at":"0001-01-01T00:00:00.000Z"}

HTTP/2.0 200 OK
Content-Length: 927
Audit-Id: 9af630c2-3fff-41ad-819f-47e8137e67e1
Content-Type: application/json
Date: Fri, 09 Nov 2018 09:06:53 GMT
Grpc-Metadata-Content-Type: application/grpc

{"id":"cd3d9ada-e3fe-11e8-96c0-42010a8000bb","name":"my-job","description":"No description provided","pipeline_spec":{"pipeline_id":"2dfffef5-dc47-4ae7-a0f4-1d23507f74c6","workflow_manifest":"{"kind":"Workflow","apiVersion":"argoproj.io/v1alpha1","metadata":{"generateName":"hello-world-","creationTimestamp":null},"spec":{"templates":[{"name":"whalesay","inputs":{},"outputs":{},"metadata":{},"container":{"name":"","image":"docker/whalesay:latest","command":["cowsay"],"args":["hello world"],"resources":{}}}],"entrypoint":"whalesay","arguments":{}},"status":{"startedAt":null,"finishedAt":null}}"},"resource_references":[{"key":{"type":"EXPERIMENT","id":"bed43f9a-fd00-4f99-a0b8-269e0941ae56"},"relationship":"OWNER"}],"max_concurrency":"1","trigger":{},"created_at":"2018-11-09T09:06:53Z","updated_at":"2018-11-09T09:06:53Z","status":"NO_STATUS","enabled":true}
created_at: "2018-11-09T09:06:53.000Z"
description: No description provided
enabled: true
id: cd3d9ada-e3fe-11e8-96c0-42010a8000bb
max_concurrency: "1"
name: my-job
pipeline_spec:
parameters: null
pipeline_id: 2dfffef5-dc47-4ae7-a0f4-1d23507f74c6
workflow_manifest: '{"kind":"Workflow","apiVersion":"argoproj.io/v1alpha1","metadata":{"generateName":"hello-world-","creationTimestamp":null},"spec":{"templates":[{"name":"whalesay","inputs":{},"outputs":{},"metadata":{},"container":{"name":"","image":"docker/whalesay:latest","command":["cowsay"],"args":["hello
world"],"resources":{}}}],"entrypoint":"whalesay","arguments":{}},"status":{"startedAt":null,"finishedAt":null}}'
resource_references:

• key:
  id: bed43f9a-fd00-4f99-a0b8-269e0941ae56
  type: EXPERIMENT
  relationship: OWNER
  status: NO_STATUS
  trigger: {}
  updated_at: "2018-11-09T09:06:53.000Z"

[vicaire]

go run main.go job list --experiment-id bed43f9a-fd00-4f99-a0b8-269e0941ae56 --debug
GET /api/v1/namespaces/kubeflow/services/ml-pipeline:8888/proxy/apis/v1beta1/jobs?page_size=10&resource_reference_key.id=bed43f9a-fd00-4f99-a0b8-269e0941ae56&resource_reference_key.type=EXPERIMENT HTTP/1.1
Host: 35.224.113.48
User-Agent: Go-http-client/1.1
Accept: application/json
Accept-Encoding: gzip

HTTP/2.0 500 Internal Server Error
Connection: close
Audit-Id: ab527a08-bd5d-4947-a9ea-20c4642cff8d
Content-Type: application/json
Date: Fri, 09 Nov 2018 09:08:14 GMT

{"error":"Failed to list jobs.: List jobs failed.: List data model failed.: InternalServerError: Failed to list jobs: sql: Scan error on column index 0, name "UUID": unsupported Scan, storing driver.Value type \u003cnil\u003e into type *string: sql: Scan error on column index 0, name "UUID": unsupported Scan, storing driver.Value type \u003cnil\u003e into type *string","code":13,"details":[{"@type":"type.googleapis.com/api.Error","error_message":"Internal Server Error","error_details":"Failed to list jobs.: List jobs failed.: List data model failed.: InternalServerError: Failed to list jobs: sql: Scan error on column index 0, name "UUID": unsupported Scan, storing driver.Value type \u003cnil\u003e into type *string: sql: Scan error on column index 0, name "UUID": unsupported Scan, storing driver.Value type \u003cnil\u003e into type *string"}]}

[vicaire]

Summary:

• The call to list jobs in an experiment does not seem to work if there is more than one job in the experiment
• It should be possible to list all the jobs, regardless of whether they have an experiment.
• Listing all the jobs in an experiment that does not exist seems to work.

There seems to be similar problems for runs:

• It should be possible to list all the runs, regardless of whether they have an experiment or job.
• Listing all the runs of an experiment return an empty array if the jobs belong to a job inside an experiment.
• Listing all the runs of a job works.

[IronPan]

it looks like you are hitting an old cluster. this is a known issue and already being fixed. Please upgrade to 0.1.2 and it should work.

[vicaire]

Thank you @IronPan. I will try again with the latest release.