Add runtime-tools' validation tests

[alban]

For now, the results of the tests are printed without returning an error.

Signed-off-by: Alban Crequy alban@kinvolk.io

---

This is superseding #1757: trying to use Travis CI instead of CircleCI.

I see the following in the logs: https://travis-ci.org/kinvolk/runc/jobs/353911723

  348 passing (57s)
  82 pending
  13 failing

[wking]

This seems overly complicated. The Makefile calling this script knows where the compiled runc is. Can't we set the RUNTIME enviroment variable in the Makefile rule?

RUNTIME=... script/oci-runtime-validation

[alban]

Good idea, I'll try that.

[alban]

I rebased this PR and this ran the tests on Travis again. The tests show the following errors:

Issues in runc:

• validation/delete.t: Failure of attempting to delete a container that is not stopped #1774
• validation/linux_rootfs_propagation_shared.t: rootfsPropagation=shared does not work #1755

Limitation of the TravisCI platform

• validation/linux_cgroups_blkio.t
• validation/linux_cgroups_relative_blkio.t

Bug in the certification tests:

• validation/linux_masked_paths.t
• validation/linux_readonly_paths.t
• validation/start.t

To investigate:

• validation/linux_rootfs_propagation_unbindable.t
• validation/linux_seccomp.t
• validation/misc_props.t
• validation/process_capabilities_fail.t

validation/delete.t ................................... 3/5 22s
  not ok attempting to `delete` a container that is not `stopped` MUST generate an error
    reference: 'https://github.com/opencontainers/runtime-spec/blob/v1.0.0/runtime.md#delete'
  
  not ok attempting to `delete` a container that is not `stopped` MUST have no effect on the container
    error: exit status 1
    reference: 'https://github.com/opencontainers/runtime-spec/blob/v1.0.0/runtime.md#delete'
    stderr: |
      container "b5f6712e-5a5e-4cdc-8e96-0032d8cfb327" does not exist

failed to create the container
container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"process_linux.go:367: setting cgroup config for procHooks process caused \\\"failed to write 8:0 500 to blkio.weight_device: write /sys/fs/cgroup/blkio/cgrouptest/blkio.weight_device: operation not supported\\\"\""
exit status 1
validation/linux_cgroups_blkio.t ...................... 0/1
  not ok validation/linux_cgroups_blkio.t
    timeout: 30000
    file: validation/linux_cgroups_blkio.t
    command: validation/linux_cgroups_blkio.t
    args: []
    stdio:
      - 0
      - pipe
      - 2
    cwd: /home/travis/gopath/src/github.com/opencontainers/runtime-tools
    exitCode: 1

failed to create the container
container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"process_linux.go:367: setting cgroup config for procHooks process caused \\\"failed to write 8:0 500 to blkio.weight_device: write /sys/fs/cgroup/blkio/testdir/cgrouptest/container/blkio.weight_device: operation not supported\\\"\""
exit status 1
validation/linux_cgroups_relative_blkio.t ............. 0/1
  not ok validation/linux_cgroups_relative_blkio.t
    timeout: 30000
    file: validation/linux_cgroups_relative_blkio.t
    command: validation/linux_cgroups_relative_blkio.t
    args: []
    stdio:
      - 0
      - pipe
      - 2
    cwd: /home/travis/gopath/src/github.com/opencontainers/runtime-tools
    exitCode: 1

validation/linux_masked_paths.t ................... 252/275
  not ok test count !== plan
    +++ found                                                           
    --- wanted                                                          
    -1                                                                  
    +274                                                                
    results:
      ok: false
      count: 274
      pass: 274
      fail: 1
      bailout: false
      todo: 0
      skip: 22
      plan:
        start: null
        end: null
        skipAll: false
        skipReason: ''
        comment: ''
      failures:
        - tapError: no plan

validation/linux_readonly_paths.t ................. 252/283
  not ok test count !== plan

failed to create the container
container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"rootfs_linux.go:46: preparing rootfs caused \\\"invalid argument\\\"\""
exit status 1
validation/linux_rootfs_propagation_shared.t ...... 252/282
  not ok shared root propogation exposes "/target672871882/mount411111073/example250057595"

failed to create the container
container_linux.go:348: starting container process caused "error adding seccomp rule for syscall personality: requested action matches default action of filter"
exit status 1
validation/linux_rootfs_propagation_unbindable.t ...... 0/1
  not ok validation/linux_rootfs_propagation_unbindable.t
    timeout: 30000
    file: validation/linux_rootfs_propagation_unbindable.t
    command: validation/linux_rootfs_propagation_unbindable.t
    args: []
    stdio:
      - 0
      - pipe
      - 2
    cwd: /home/travis/gopath/src/github.com/opencontainers/runtime-tools
    exitCode: 1

validation/linux_seccomp.t ............................ 0/1
  not ok validation/linux_seccomp.t
    timeout: 30000
    file: validation/linux_seccomp.t
    command: validation/linux_seccomp.t
    args: []
    stdio:
      - 0
      - pipe
      - 2
    cwd: /home/travis/gopath/src/github.com/opencontainers/runtime-tools
    exitCode: 1

validation/misc_props.t ............................... 2/3 3s
  not ok runtimes that are reading or processing this configuration file MUST generate an error when invalid or unsupported values are encountered
    reference: >-
      https://github.com/opencontainers/runtime-spec/blob/v1.0.0/config.md#valid-values

Any value which cannot be mapped to a relevant kernel interface MUST cause an error
Refer to: https://github.com/opencontainers/runtime-spec/blob/v1.0.0/config.md#linux-process
validation/process_capabilities_fail.t ............ 252/288
  not ok validation/process_capabilities_fail.t
    timeout: 30000
    file: validation/process_capabilities_fail.t
    command: validation/process_capabilities_fail.t
    args: []
    stdio:
      - 0
      - pipe
      - 2
    cwd: /home/travis/gopath/src/github.com/opencontainers/runtime-tools
    exitCode: 1

exit status 2
validation/start.t .................................... 6/7
  not ok test count !== plan

[alban]

I have run the tests again on my laptop:

Issues in runc:

• non-path source mounts (proc, sysfs). PR needs merge: Fix regression with mounts with non-absolute source path #1845
• delete unstopped containers: issue: Failure of attempting to delete a container that is not stopped #1774
• shared propagation rootfs: PR needs review: rootfsPropagation=shared does not work #1755
• unbindable propagation rootfs: we will have to look into that after rootfsPropagation=shared does not work #1755
• no error on invalid values: issue: runc does not error on ociVersion=invalid #1846
• cgroup namespace: PR Carry #781: Add support for cgroup namespace #1184
• seccomp: issue: seccomp rejects redundant rules with "requested action matches default action of filter" #1847
• start: partially fixed by PR Fix panic with config.json with empty process #1848
• hooks_stdin: libcontainer: Set 'status' in hook stdin #1741 and 'prestart/poststart' hooks are done in the 'create' operation #1710

[AkihiroSuda]

What's current status?

[AkihiroSuda]

ping

[alban]

Obsolete patch: the CI significantly changed since 2018.