format-coverage fails for simplecov 0.21.2 (minitest) with cc-test-reporter 0.10.3

[tnir]

What happens

cc-test-reporter format-coverage fails with coverage results from SimpleCov 0.21.2 and minitest 5.16.1 as follows:

$ ./cc-test-reporter format-coverage -t simplecov coverage/.resultset.json -d
DEBU[0000] coverage path coverage/.resultset.json       
DEBU[0000] using formatter simplecov                    
DEBU[0000] checking search path coverage/.resultset.json for simplecov formatter 
DEBU[0000] couldn't load branch from ENV, trying git... 
DEBU[0000] couldn't load commit sha from ENV, trying git... 
DEBU[0000] couldn't load committed at from ENV, trying git... 
DEBU[0000] Analyzing simplecov json output from latest format coverage/.resultset.json 
DEBU[0000] Analyzing simplecov json output from legacy format coverage/.resultset.json 
Error: json: cannot unmarshal object into Go struct field resultSet.coverage of type []formatters.NullInt

Why

The following results should be handled without any error:

{
  "Unit Tests": {
    "coverage": {
      "/workspace/geminabox/lib/geminabox.rb": {
        "lines": [
          null
        ]
      },
      "/workspace/geminabox/lib/geminabox/proxy.rb": {
        "lines": [
          null
        ]
      }
    },
    "timestamp": 1650000000
  },
  "Integration Tests": {
    "coverage": {
      "/workspace/geminabox/lib/geminabox.rb": {
        "lines": [
          null
        ]
      },
      "/workspace/geminabox/lib/geminabox/proxy.rb": {
        "lines": [
          null
        ]
      }
    },
    "timestamp": 1650000000
  }
}

Currently it work well with the following results:

{
  "Unit Tests": {
    "coverage": {
      "/workspace/geminabox/lib/geminabox.rb": [
          null
      ],
      "/workspace/geminabox/lib/geminabox/proxy.rb": [
          null
      ]
    },
    "timestamp": 1650000000
  },
  "Integration Tests": {
    "coverage": {
      "/workspace/geminabox/lib/geminabox.rb": [
          null
      ],
      "/workspace/geminabox/lib/geminabox/proxy.rb": [
          null
      ]
    },
    "timestamp": 1650000000
  }
}

Workaround

jq 'map_values(. | map_values(if type=="object" then map_values(.lines) else . end))' coverage/.resultset.json helped me work well.

Environments

Code Climate Test Reporter 0.10.3 (2cd3a82f34475dd7387da166ddd967fe1d713a65 @ 2022-02-15T15:28:53+0000)

• simplecov (0.21.2)
• simplecov_json_formatter (0.1.4)
• minitest (5.16.1)
• Send coverage report to CodeClimate geminabox/geminabox#475

[tnir]

Looks relevant to #488 to me.

• Make simplecov json formatter more flexible #488

[manicmaniac]

@tnir

Thank you for the workaround in #495 (comment)

📝 jq 'walk(try .lines // .)' coverage/.resultset.json also works for me.

[manicmaniac]

Currently simplecov formatter checks .meta.simplecov_version value in .resultset.json and changes its behavior depending whether the version is above 0.18.0.

However, apparently simplecov does not output .meta.simplecov_version to .resultset.json as far as I confirmed.
If .meta.simplecov_version does not exist, the program uses legacy formatter that does not support .lines.

test-reporter/formatters/simplecov/json_formatter.go

Line 85 in 22de46a

if reportIsOnLegacyFormat(m.Meta.SimpleCovVersion) {

test-reporter/formatters/simplecov/json_formatter.go

Lines 35 to 46 in 22de46a

	func reportIsOnLegacyFormat(simplecovVersion string) bool {
	var major, minor, patch int
	fmt.Sscanf(simplecovVersion, "%d.%d.%d", &major, &minor, &patch)
	if major < 1 {
	if minor < 18 {
	return true
	}
	}
	return false
	}

[manicmaniac]

However, apparently simplecov does not output .meta.simplecov_version to .resultset.json as far as I confirmed.

After a further investigation, I found simplecov writes coverage/coverage.json only when CC_TEST_REPORTER_ID environment variable is set (this feature was introduced in simplecov-ruby/simplecov#950).
And cc-test-reporter format-coverage --input-type=simplecov reads coverage/coverage.json at first and fallbacks to coverage/.resultset.json if the former does not exist.

test-reporter/formatters/simplecov/simplecov.go

Line 12 in 22de46a

var searchPaths = []string{"coverage/coverage.json", "coverage/.resultset.json"}

So another possible workaround would be setting environment variable before testing.

CC_TEST_REPORTER_ID=xxx bundle exec rake test