Skip to content

Commit

Permalink
Rollup merge of #83755 - richkadel:cov-test-simplify, r=tmandry
Browse files Browse the repository at this point in the history
Simplify coverage tests

This change reduces the risk of impacting coverage tests on unrelated
changes (such as MIR and Span changes), and reduces the burden when
blessing coverage changes in case it is necessary.

* Remove all spanview tests. The spanview tests were useful during
  development, but they can be generated as needed, via compiler command
  line flags. They aren't critical to confirming coverage results. (The
  coverage report tests are sufficient.)

  When spanview regeneration was necessary, the diffs were way too hard
  to read to be useful anyway. So I'm removing them to reduce friction
  from a feature that is no longer useful.

* Remove the requirement for `llvm-cov show --debug` when blessing
  tests. The `--debug` flag is, unfortunately, only available if LLVM is
  built with `optimize = false` (in Rust's config.toml). This adds
  significant time and resource burdens to the contributor's build. As
  it turns out, for other reasons in the past, I wasn't actually using
  the debug output (counter info) to validate coverage anymore either,
  so it was required for no reason, I now realize.
  • Loading branch information
Dylan-DPC authored Apr 2, 2021
2 parents 6cb74ad + fad5388 commit 7009117
Show file tree
Hide file tree
Showing 131 changed files with 0 additions and 26,664 deletions.
80 changes: 0 additions & 80 deletions src/test/run-make-fulldeps/coverage-reports/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -61,13 +61,6 @@ endif
LLVM_COV_IGNORE_FILES=\
--ignore-filename-regex='(uses_crate.rs|uses_inline_crate.rs)'

# When generating `expected_*` results (using `x.py test --bless`), the `--debug` flag is forced.
# If assertions are disabled, the command will fail with an error, rather than attempt to generate
# only partial results.
ifdef RUSTC_BLESS_TEST
DEBUG_FLAG=--debug
endif

all: $(patsubst $(SOURCEDIR)/lib/%.rs,%,$(wildcard $(SOURCEDIR)/lib/*.rs)) $(patsubst $(SOURCEDIR)/%.rs,%,$(wildcard $(SOURCEDIR)/*.rs))

# Ensure there are no `expected` results for tests that may have been removed or renamed
Expand Down Expand Up @@ -177,76 +170,3 @@ else
false \
)
endif

####################################################################################################

# The following Makefile content was used to copy the generated `counters` files
# to `expected_` files (when `--bless`ed) and to compare them via `diff`; but for
# multiple reasons, these files cannot easily be used for test validation:
#
# * Output lines can be produced in non-deterministic order (depending on the
# target platform, and sometimes on unrelated codegen changes).
# * Some lines include demangled function names, making them more challenging
# to interpret and compare.
#
# The files are still generated (in `$(TMPDIR)`) to support developers wanting
# to inspect the counters, for debugging purposes.
#
# ifdef RUSTC_BLESS_TEST
# cp "$(TMPDIR)"/actual_show_coverage_counters.$@.txt \
# expected_show_coverage_counters.$@.txt
# else
#
# ifdef DEBUG_FLAG
# $(DIFF) expected_show_coverage_counters.$@.txt "$(TMPDIR)"/actual_show_coverage_counters.$@.txt || \
# ( grep -q '^\/\/ ignore-llvm-cov-show-diffs' $(SOURCEDIR)/$@.rs && \
# >&2 echo 'diff failed, but suppressed with `// ignore-llvm-cov-show-diffs` in $(SOURCEDIR)/$@.rs' \
# ) || \
# ( >&2 echo 'diff failed, and not suppressed without `// ignore-llvm-cov-show-diffs` in $(SOURCEDIR)/$@.rs'; \
# >&2 echo '(Ignore anyway until mangled function names in "counters" files are demangled.)' \
# )
# endif
#
# endif

####################################################################################################

# The following Makefile content, and short JSON script, were used to generate
# coverage reports in JSON when the `llvm-cov show` reports were less reliable for
# testing. At the present time, however, the `llvm-cov show` results, and methods
# for comparing them, are working for all tests, making the JSON reports redundant.
#
# If this changes in the future, the scripts are left here, commented out, but can
# be resurrected if desired. This could be used to compare *only* the JSON files;
# and in that case, the `llvm-cov show` reports can be ignored by inserting
# `// ignore-llvm-cov-show-diffs` at the top of the source file.
#
# # Generate a coverage report in JSON, using `llvm-cov export`, and fail if
# # there are differences from the expected output.
# "$(LLVM_BIN_DIR)"/llvm-cov export \
# $(LLVM_COV_IGNORE_FILES) \
# --summary-only \
# --instr-profile="$(TMPDIR)"/$@.profdata \
# $(call BIN,"$(TMPDIR)"/$@) \
# | "$(PYTHON)" $(BASEDIR)/prettify_json.py \
# > "$(TMPDIR)"/actual_export_coverage.$@.json
#
# ifdef RUSTC_BLESS_TEST
# cp "$(TMPDIR)"/actual_export_coverage.$@.json expected_export_coverage.$@.json
# else
# # Check that exported JSON coverage data matches what we expect (`--bless` refreshes `expected`)
# $(DIFF) expected_export_coverage.$@.json "$(TMPDIR)"/actual_export_coverage.$@.json
# endif
#
# # # If generating coverage reports in JSON, this Makefile is accompanied by
# # # a Python script, `prettify_json.py`, which is defined:
# #
# # #!/usr/bin/env python
# #
# # import sys
# # import json
# #
# # # Try to decode line in order to ensure it is a valid JSON document
# # for line in sys.stdin:
# # parsed = json.loads(line)
# # print (json.dumps(parsed, indent=2, separators=(',', ': '), sort_keys=True))
92 changes: 0 additions & 92 deletions src/test/run-make-fulldeps/coverage-spanview/Makefile

This file was deleted.

12 changes: 0 additions & 12 deletions src/test/run-make-fulldeps/coverage-spanview/escape_url.py

This file was deleted.

This file was deleted.

Loading

0 comments on commit 7009117

Please sign in to comment.