Fix flaky tests. Use jq to sort JSON arrays before comparing them.

[mindaugasrukas]

PR Checklist

Please check if your PR fulfills the following requirements:

• Tests for the changes have been added (for bug fixes / features)
• Docs have been added / updated (for bug fixes / features)

PR Type

What kind of change does this PR introduce?

• Bugfix
• Feature
• Code style update (formatting, local variables)
• Refactoring (no functional changes, no api changes)
• Build related changes
• CI related changes
• Documentation content changes
• Other... Please describe:

Flaky tests.

What is the current behavior?

Tests fail due to JSON array order:

$ ./testing/e2e.sh test_new_container_push_skip_unchanged_digest_changed
Checking INFO: Invocation ID: 9a49599a-3794-414f-86ce-7f1d08eac3de
Loading:
Loading: 0 packages loaded
DEBUG: /home/user/.cache/bazel/_bazel_user/5e184aea21a9eb5699227e7c322d8bc1/external/bazel_toolchains/rules/rbe_repo/version_check.bzl:68:14:
Current running Bazel is ahead of bazel-toolchains repo. Please update your pin to bazel-toolchains repo in your WORKSPACE file.
DEBUG: /home/user/.cache/bazel/_bazel_user/5e184aea21a9eb5699227e7c322d8bc1/external/bazel_toolchains/rules/rbe_repo/checked_in.bzl:125:14: buildkite_config not using checked in configs; Bazel version 4.2.1 was picked/selected but no checked in config was found in map {"0.20.0": ["8.0.0"], "0.21.0": ["8.0.0"], "0.22.0": ["8.0.0", "9.0.0"], "0.23.0": ["8.0.0", "9.0.0"], "0.23.1": ["8.0.0", "9.0.0"], "0.23.2": ["9.0.0"], "0.24.0": ["9.0.0"], "0.24.1": ["9.0.0"], "0.25.0": ["9.0.0"], "0.25.1": ["9.0.0"], "0.25.2": ["9.0.0"], "0.26.0": ["9.0.0"], "0.26.1": ["9.0.0"], "0.27.0": ["9.0.0"], "0.27.1": ["9.0.0"], "0.28.0": ["9.0.0"], "0.28.1": ["9.0.0"], "0.29.0": ["9.0.0"], "0.29.1": ["9.0.0", "10.0.0"], "1.0.0": ["9.0.0", "10.0.0"], "1.0.1": ["10.0.0"], "1.1.0": ["10.0.0"], "1.2.0": ["10.0.0"], "1.2.1": ["10.0.0"], "2.0.0": ["10.0.0"], "2.1.0": ["10.0.0"], "2.1.1": ["10.0.0", "11.0.0"], "2.2.0": ["11.0.0"], "3.0.0": ["11.0.0"], "3.1.0": ["11.0.0"], "3.2.0": ["11.0.0"], "3.3.0": ["11.0.0"], "3.3.1": ["11.0.0"], "3.4.1": ["11.0.0"], "3.5.0": ["11.0.0"], "3.5.1": ["11.0.0"], "3.6.0": ["11.0.0"], "3.7.0": ["11.0.0"], "3.7.1": ["11.0.0"], "3.7.2": ["11.0.0"], "4.0.0": ["11.0.0"]}
Analyzing: target //tests/container:new_push_test_skip_unchanged_digest_changed_tag_1 (0 packages loaded, 0 targets configured)
DEBUG: /home/user/bazelbuild/rules_docker/container/push.bzl:80:14: Pushing an image based on a tarball can be very expensive. If the image set on //tests/container:new_push_test_skip_unchanged_digest_changed_tag_1 is the output of a container_build, consider dropping the '.tar' extension. If the image is checked in, consider using container_import instead.
INFO: Analyzed target //tests/container:new_push_test_skip_unchanged_digest_changed_tag_1 (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
[0 / 1] [Prepa] BazelWorkspaceStatusAction stable-status.txt
Target //tests/container:new_push_test_skip_unchanged_digest_changed_tag_1 up-to-date:
  bazel-bin/tests/container/new_push_test_skip_unchanged_digest_changed_tag_1.digest
  bazel-bin/tests/container/new_push_test_skip_unchanged_digest_changed_tag_1
INFO: Elapsed time: 0.188s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/tests/container/new_push_test_skip_unchanged_digest_changed_tag_1
INFO: Build completed successfully, 1 total action
2022/01/19 00:55:33 Error checking if digest already exists unable to get remote image: GET http://localhost:5000/v2/docker/test/manifests/sha256:e35109210dedbb7c9e333868e5351634d634372f9e708b6999086303319cfdf8: MANIFEST_UNKNOWN: manifest unknown; map[Name:docker/test Revision:sha256:e35109210dedbb7c9e333868e5351634d634372f9e708b6999086303319cfdf8]. Still pushing
2022/01/19 00:55:33 Successfully pushed Docker image to localhost:5000/docker/test:changed_tag1 contains Successfully pushed Docker image
Checking INFO: Invocation ID: 10d973ed-00a8-4c01-a919-3faabd99c51b
Loading:
Loading: 0 packages loaded
DEBUG: /home/user/.cache/bazel/_bazel_user/5e184aea21a9eb5699227e7c322d8bc1/external/bazel_toolchains/rules/rbe_repo/version_check.bzl:68:14:
Current running Bazel is ahead of bazel-toolchains repo. Please update your pin to bazel-toolchains repo in your WORKSPACE file.
DEBUG: /home/user/.cache/bazel/_bazel_user/5e184aea21a9eb5699227e7c322d8bc1/external/bazel_toolchains/rules/rbe_repo/checked_in.bzl:125:14: buildkite_config not using checked in configs; Bazel version 4.2.1 was picked/selected but no checked in config was found in map {"0.20.0": ["8.0.0"], "0.21.0": ["8.0.0"], "0.22.0": ["8.0.0", "9.0.0"], "0.23.0": ["8.0.0", "9.0.0"], "0.23.1": ["8.0.0", "9.0.0"], "0.23.2": ["9.0.0"], "0.24.0": ["9.0.0"], "0.24.1": ["9.0.0"], "0.25.0": ["9.0.0"], "0.25.1": ["9.0.0"], "0.25.2": ["9.0.0"], "0.26.0": ["9.0.0"], "0.26.1": ["9.0.0"], "0.27.0": ["9.0.0"], "0.27.1": ["9.0.0"], "0.28.0": ["9.0.0"], "0.28.1": ["9.0.0"], "0.29.0": ["9.0.0"], "0.29.1": ["9.0.0", "10.0.0"], "1.0.0": ["9.0.0", "10.0.0"], "1.0.1": ["10.0.0"], "1.1.0": ["10.0.0"], "1.2.0": ["10.0.0"], "1.2.1": ["10.0.0"], "2.0.0": ["10.0.0"], "2.1.0": ["10.0.0"], "2.1.1": ["10.0.0", "11.0.0"], "2.2.0": ["11.0.0"], "3.0.0": ["11.0.0"], "3.1.0": ["11.0.0"], "3.2.0": ["11.0.0"], "3.3.0": ["11.0.0"], "3.3.1": ["11.0.0"], "3.4.1": ["11.0.0"], "3.5.0": ["11.0.0"], "3.5.1": ["11.0.0"], "3.6.0": ["11.0.0"], "3.7.0": ["11.0.0"], "3.7.1": ["11.0.0"], "3.7.2": ["11.0.0"], "4.0.0": ["11.0.0"]}
Analyzing: target //tests/container:new_push_test_skip_unchanged_digest_changed_tag_2 (0 packages loaded, 0 targets configured)
DEBUG: /home/user/bazelbuild/rules_docker/container/push.bzl:80:14: Pushing an image based on a tarball can be very expensive. If the image set on //tests/container:new_push_test_skip_unchanged_digest_changed_tag_2 is the output of a container_build, consider dropping the '.tar' extension. If the image is checked in, consider using container_import instead.
INFO: Analyzed target //tests/container:new_push_test_skip_unchanged_digest_changed_tag_2 (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
[0 / 1] [Prepa] BazelWorkspaceStatusAction stable-status.txt
Target //tests/container:new_push_test_skip_unchanged_digest_changed_tag_2 up-to-date:
  bazel-bin/tests/container/new_push_test_skip_unchanged_digest_changed_tag_2.digest
  bazel-bin/tests/container/new_push_test_skip_unchanged_digest_changed_tag_2
INFO: Elapsed time: 0.183s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/tests/container/new_push_test_skip_unchanged_digest_changed_tag_2
INFO: Build completed successfully, 1 total action
2022/01/19 00:55:33 Error checking if digest already exists unable to get remote image: GET http://localhost:5000/v2/docker/test/manifests/sha256:c214132ea116a93c241d3f8e5c7275bd355ed6ebebcd4d25cbc7d63882301e60: MANIFEST_UNKNOWN: manifest unknown; map[Name:docker/test Revision:sha256:c214132ea116a93c241d3f8e5c7275bd355ed6ebebcd4d25cbc7d63882301e60]. Still pushing
2022/01/19 00:55:33 Successfully pushed Docker image to localhost:5000/docker/test:changed_tag2 contains Successfully pushed Docker image
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    62  100    62    0     0  31000      0 --:--:-- --:--:-- --:--:-- 31000
Checking {"name":"docker/test","tags":["changed_tag2","changed_tag1"]} contains {"name":"docker/test","tags":["changed_tag1","changed_tag2"]}
FAILURE: Expected '{"name":"docker/test","tags":["changed_tag1","changed_tag2"]}' not found in '{"name":"docker/test","tags":["changed_tag2","changed_tag1"]}'

Issue Number: #1997

What is the new behavior?

Tests pass:

$ ./testing/e2e.sh test_new_container_push_skip_unchanged_digest_changed
Checking INFO: Invocation ID: 18d85c8d-52dd-4564-9a4a-d9a521a6f929
Loading:
Loading: 0 packages loaded
DEBUG: /home/user/.cache/bazel/_bazel_user/5e184aea21a9eb5699227e7c322d8bc1/external/bazel_toolchains/rules/rbe_repo/version_check.bzl:68:14:
Current running Bazel is ahead of bazel-toolchains repo. Please update your pin to bazel-toolchains repo in your WORKSPACE file.
DEBUG: /home/user/.cache/bazel/_bazel_user/5e184aea21a9eb5699227e7c322d8bc1/external/bazel_toolchains/rules/rbe_repo/checked_in.bzl:125:14: buildkite_config not using checked in configs; Bazel version 4.2.1 was picked/selected but no checked in config was found in map {"0.20.0": ["8.0.0"], "0.21.0": ["8.0.0"], "0.22.0": ["8.0.0", "9.0.0"], "0.23.0": ["8.0.0", "9.0.0"], "0.23.1": ["8.0.0", "9.0.0"], "0.23.2": ["9.0.0"], "0.24.0": ["9.0.0"], "0.24.1": ["9.0.0"], "0.25.0": ["9.0.0"], "0.25.1": ["9.0.0"], "0.25.2": ["9.0.0"], "0.26.0": ["9.0.0"], "0.26.1": ["9.0.0"], "0.27.0": ["9.0.0"], "0.27.1": ["9.0.0"], "0.28.0": ["9.0.0"], "0.28.1": ["9.0.0"], "0.29.0": ["9.0.0"], "0.29.1": ["9.0.0", "10.0.0"], "1.0.0": ["9.0.0", "10.0.0"], "1.0.1": ["10.0.0"], "1.1.0": ["10.0.0"], "1.2.0": ["10.0.0"], "1.2.1": ["10.0.0"], "2.0.0": ["10.0.0"], "2.1.0": ["10.0.0"], "2.1.1": ["10.0.0", "11.0.0"], "2.2.0": ["11.0.0"], "3.0.0": ["11.0.0"], "3.1.0": ["11.0.0"], "3.2.0": ["11.0.0"], "3.3.0": ["11.0.0"], "3.3.1": ["11.0.0"], "3.4.1": ["11.0.0"], "3.5.0": ["11.0.0"], "3.5.1": ["11.0.0"], "3.6.0": ["11.0.0"], "3.7.0": ["11.0.0"], "3.7.1": ["11.0.0"], "3.7.2": ["11.0.0"], "4.0.0": ["11.0.0"]}
Analyzing: target //tests/container:new_push_test_skip_unchanged_digest_changed_tag_1 (0 packages loaded, 0 targets configured)
DEBUG: /home/user/bazelbuild/rules_docker/container/push.bzl:80:14: Pushing an image based on a tarball can be very expensive. If the image set on //tests/container:new_push_test_skip_unchanged_digest_changed_tag_1 is the output of a container_build, consider dropping the '.tar' extension. If the image is checked in, consider using container_import instead.
INFO: Analyzed target //tests/container:new_push_test_skip_unchanged_digest_changed_tag_1 (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
[0 / 1] [Prepa] BazelWorkspaceStatusAction stable-status.txt
Target //tests/container:new_push_test_skip_unchanged_digest_changed_tag_1 up-to-date:
  bazel-bin/tests/container/new_push_test_skip_unchanged_digest_changed_tag_1.digest
  bazel-bin/tests/container/new_push_test_skip_unchanged_digest_changed_tag_1
INFO: Elapsed time: 0.182s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/tests/container/new_push_test_skip_unchanged_digest_changed_tag_1
INFO: Build completed successfully, 1 total action
2022/01/19 00:57:13 Error checking if digest already exists unable to get remote image: GET http://localhost:5000/v2/docker/test/manifests/sha256:e35109210dedbb7c9e333868e5351634d634372f9e708b6999086303319cfdf8: MANIFEST_UNKNOWN: manifest unknown; map[Name:docker/test Revision:sha256:e35109210dedbb7c9e333868e5351634d634372f9e708b6999086303319cfdf8]. Still pushing
2022/01/19 00:57:13 Successfully pushed Docker image to localhost:5000/docker/test:changed_tag1 contains Successfully pushed Docker image
Checking INFO: Invocation ID: 7fe46ef6-a21a-4da1-9dd2-9c42db8439f5
Loading:
Loading: 0 packages loaded
DEBUG: /home/user/.cache/bazel/_bazel_user/5e184aea21a9eb5699227e7c322d8bc1/external/bazel_toolchains/rules/rbe_repo/version_check.bzl:68:14:
Current running Bazel is ahead of bazel-toolchains repo. Please update your pin to bazel-toolchains repo in your WORKSPACE file.
DEBUG: /home/user/.cache/bazel/_bazel_user/5e184aea21a9eb5699227e7c322d8bc1/external/bazel_toolchains/rules/rbe_repo/checked_in.bzl:125:14: buildkite_config not using checked in configs; Bazel version 4.2.1 was picked/selected but no checked in config was found in map {"0.20.0": ["8.0.0"], "0.21.0": ["8.0.0"], "0.22.0": ["8.0.0", "9.0.0"], "0.23.0": ["8.0.0", "9.0.0"], "0.23.1": ["8.0.0", "9.0.0"], "0.23.2": ["9.0.0"], "0.24.0": ["9.0.0"], "0.24.1": ["9.0.0"], "0.25.0": ["9.0.0"], "0.25.1": ["9.0.0"], "0.25.2": ["9.0.0"], "0.26.0": ["9.0.0"], "0.26.1": ["9.0.0"], "0.27.0": ["9.0.0"], "0.27.1": ["9.0.0"], "0.28.0": ["9.0.0"], "0.28.1": ["9.0.0"], "0.29.0": ["9.0.0"], "0.29.1": ["9.0.0", "10.0.0"], "1.0.0": ["9.0.0", "10.0.0"], "1.0.1": ["10.0.0"], "1.1.0": ["10.0.0"], "1.2.0": ["10.0.0"], "1.2.1": ["10.0.0"], "2.0.0": ["10.0.0"], "2.1.0": ["10.0.0"], "2.1.1": ["10.0.0", "11.0.0"], "2.2.0": ["11.0.0"], "3.0.0": ["11.0.0"], "3.1.0": ["11.0.0"], "3.2.0": ["11.0.0"], "3.3.0": ["11.0.0"], "3.3.1": ["11.0.0"], "3.4.1": ["11.0.0"], "3.5.0": ["11.0.0"], "3.5.1": ["11.0.0"], "3.6.0": ["11.0.0"], "3.7.0": ["11.0.0"], "3.7.1": ["11.0.0"], "3.7.2": ["11.0.0"], "4.0.0": ["11.0.0"]}
Analyzing: target //tests/container:new_push_test_skip_unchanged_digest_changed_tag_2 (0 packages loaded, 0 targets configured)
DEBUG: /home/user/bazelbuild/rules_docker/container/push.bzl:80:14: Pushing an image based on a tarball can be very expensive. If the image set on //tests/container:new_push_test_skip_unchanged_digest_changed_tag_2 is the output of a container_build, consider dropping the '.tar' extension. If the image is checked in, consider using container_import instead.
INFO: Analyzed target //tests/container:new_push_test_skip_unchanged_digest_changed_tag_2 (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
[0 / 1] [Prepa] BazelWorkspaceStatusAction stable-status.txt
Target //tests/container:new_push_test_skip_unchanged_digest_changed_tag_2 up-to-date:
  bazel-bin/tests/container/new_push_test_skip_unchanged_digest_changed_tag_2.digest
  bazel-bin/tests/container/new_push_test_skip_unchanged_digest_changed_tag_2
INFO: Elapsed time: 0.174s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/tests/container/new_push_test_skip_unchanged_digest_changed_tag_2
INFO: Build completed successfully, 1 total action
2022/01/19 00:57:14 Error checking if digest already exists unable to get remote image: GET http://localhost:5000/v2/docker/test/manifests/sha256:c214132ea116a93c241d3f8e5c7275bd355ed6ebebcd4d25cbc7d63882301e60: MANIFEST_UNKNOWN: manifest unknown; map[Name:docker/test Revision:sha256:c214132ea116a93c241d3f8e5c7275bd355ed6ebebcd4d25cbc7d63882301e60]. Still pushing
2022/01/19 00:57:14 Successfully pushed Docker image to localhost:5000/docker/test:changed_tag2 contains Successfully pushed Docker image
Checking {"name":"docker/test","tags":["changed_tag1","changed_tag2"]} contains {"name":"docker/test","tags":["changed_tag1","changed_tag2"]}

Does this PR introduce a breaking change?

• Yes
• No

It adds the jq dependency.

[pcj]

Thanks @mindaugasrukas for taking at look at this (and your related e2e PRs).

Is jq in platform: ubuntu1804?

[mindaugasrukas]

Thanks @mindaugasrukas for taking at look at this (and your related e2e PRs).

Is jq in platform: ubuntu1804?

It is not.

However, as this is affecting only the E2E tests, and they are still not executed during CI, I think it's fine as the CI prerun command could be expanded to: apt-get install jq -y.