Better TEST_OUTPUT

[dkorpel]

When you have a big fail_compilation test, for example:

https://github.com/dlang/dmd/blob/master/compiler/test/fail_compilation/retscope6.d

There are two annoyances:

• Because of multiple TEST_OUTPUT blocks, they cannot be automatically updated with AUTO_UPDATE=1
• Because of #line directives for stable lines, you cannot quickly locate an error's corresponding line in your text editor.

(See also #14434 (comment) and #14434 (comment) where this came up)

Splitting it up into smaller tests isn't all so great either:

• The huge amount of test files is annoying to browse. The GitHub web viewer even truncates it to the first 1000.
• Every file gives a bit of overhead making the test suite run slower

But what if instead of TEST_OUTPUT blocks, the test runner could also parse // Error: or // Deprecation: comments:

int* test() @safe
{
    int i;
    int*[][] arr = new int*[][](1);
// Error: copying `& i` into allocated memory escapes a reference to local variable `i`
    arr[0] ~= &i;
    return arr[0][0];
}

The whole fail_compilation/retscope6.d(6007): prefix will be inferred from the location of the comment in the source file. I think this is both easy to write, read, and mechanically auto update. What do you think?

[Geod24]

I think it should be marked with some kind of unmistakable marker (e.g. @@TEST@@) so that you can still copy-paste old error message without having false positive, but it's not a big point either. I'm all for making testing better and it looks like this would.

[ibuclaw]

For comparison, this is an example of marking errors in dejagnu (also, if an error occurs twice on the same line).
https://github.com/D-Programming-GDC/gcc/blob/ci/mainline/gcc/testsuite/gdc.dg/asm1.d#L8-L19

[maxhaton]

https://github.com/ldc-developers/ldc/blob/master/tests/fail_compilation/betterC_typeinfo_diag.d#L8 ldc/llvm equiv

[ljmf00-wekaio]

Why don't we use lit with FileCheck? LLVM does this for years and it's much more intuitive. See https://llvm.org/docs/CommandGuide/FileCheck.html

One example:

; RUN: llvm-as < %s | llc -mtriple=i686-apple-darwin9 -mattr=sse41 \
; RUN:              | FileCheck %s -check-prefix=X32
; RUN: llvm-as < %s | llc -mtriple=x86_64-apple-darwin9 -mattr=sse41 \
; RUN:              | FileCheck %s -check-prefix=X64

define <4 x i32> @pinsrd_1(i32 %s, <4 x i32> %tmp) nounwind {
        %tmp1 = insertelement <4 x i32>; %tmp, i32 %s, i32 1
        ret <4 x i32> %tmp1
; X32: pinsrd_1:
; X32:    pinsrd $1, 4(%esp), %xmm0

; X64: pinsrd_1:
; X64:    pinsrd $1, %edi, %xmm0
}

[ljmf00-wekaio]

FileCheck works for everything that has an output, its not tied to any specific output standard. Its pure pattern matching checks. So we can add tests to the output of -vasm e.g. or other things, like dwarf backend implementation.
I see dwarf tests particularly janky, when I did changes there.

[ljmf00-wekaio]

CC @JohanEngelen @kinke maybe you can help me supporting this idea with suggestions over our current testsuite in DMD :)

[JohanEngelen]

lit and FileCheck and not (and split-file) that LDC uses are way better than the current dmd testsuite. Fully support converting to that.

[dkorpel]

much more intuitive

Idk, that example you gave pretty much looks like gibberish to me 😆

But I do like the idea of using a common tool instead of an own invention, so it's worth a look at.

[ljmf00]

Idk, that example you gave pretty much looks like gibberish to me 😆

Probably because its matching assembly but its pretty intuitive if you look at the link I sent.