In language test support

[smoothdeveloper]

This is a broader suggestion related to testing and software quality, right in the core of the language, than a specific feature.

in language test support:

As prior art, languages such as Cobra and Rust support concept of "test blocks":

• Cobra: http://cobra-language.com/ http://cobra-language.com/trac/cobra/wiki/Tests
• Rust: https://www.rust-lang.org/ https://doc.rust-lang.org/book/ch11-01-writing-tests.html

design by contract

• Effeil (https://www.eiffel.org/) and Cobra implements "design by contract" with requireand ensure blocks, which are great way to encode invariants in asserts (I assume the constructs may be useful for "logical" linear type analyzer kind of stuff, that could parse those blocks and do flow analysis).

verifiable interpreter output

See doctest in Python and Haskell:

• https://docs.python.org/3/library/doctest.html
• https://serokell.io/blog/doctests-in-haskell

For this, I'd actually prefer we devise something which refer to free standing .fsx and their "golden output", a bit like some of our baseline driven test infrastructure.

Having all those things, baked in the language, may have a significant impact on the quality of codebases built on it, by establishing clear guidelines, to make important code humanely maintainable where it matters.

more ideas

open for people to share more things they've come across or thought of, in context of language test support.

Pros

• zeroes cost of context switch, with things like picking a test runner, creating and managing more fsproj in a solution
• F# join the set languages where this is "done right"
• more libraries with more invariants that may invariably hold

Cons

• tons of work, especially to integrate in comprehensive API in FCS
• people may not embrace those specific approaches as being worthy of specific attention in the language

Extra information

Estimated cost: XXL

Please tick all that apply:

• This is not a breaking change to the F# language design
• I or my company would be willing to help implement and/or test this

[charlesroddie]

The main problem I find with testing in dotnet is lack of type safety in the test constructs. It is annotation-based and the interface for data-driven tests is particularly bad. These are the terrible lengths that we go to get a minimal degree of safety:

let (someTest', someData) = // someData is part of the boilerplate and is a nongeneric TestCaseData list
    let test = ... // 'a -> unit
    let annotatedData = ... // seq<string * 'a>
    Test.GetTestWithData(test, annotatedData) // type-safe inputs

// This is the main boilerplate
[<TestCaseSource(nameof(someData))>] // data is referenced as a string which is not great
let someTest args = someTest' args // there is no guarantee that the type of args matches someData, but if you stick to the boilerplate template exactly then safety is guaranteed by the type-safe part of the code

What would be good is to have a type-safe dotnet test API, which maintains the convenience of existing test runners (which have desktop IDE UIs for running individual tests and groups of tests and reporting test failures, and web UIs for CI). This would define tests in proper code (rather than annotation/reflection-based scanning), running the code to get a list of tests that can be run. The API could look like this:

namespace TestFramework

open System.Collections.Generic

module Assert =
    let isTrue(b:bool) = if not b then failwith "false"

type Test(name:string, test:unit -> unit) =
    member _.Name = name
    member _.Test = test

type TestCase<'a>(name:string, data:'a) =
    member _.Name = name
    member _.Data = data

type TestFolder(name:string, subfolders: IReadOnlyCollection<TestFolder>, tests: IReadOnlyCollection<Test>) =
    member _.Name = name
    member _.Subfolders = subfolders
    member _.Tests = tests
    static member FromData<'a>(name:string, data: IReadOnlyCollection<TestCase<'a>>, test:'a -> unit) =
        TestFolder(name, [], [ for tc in data do yield Test(tc.Name, fun () -> test tc.Data) ])

type TestRunner =
    static member Load(tf:TestFolder) = ... // Loads a test folder into a test runner UI, from which all items, some items may be run

// Example usage

module TestUsageExample

let additions =
    let tests = [
        Test("1+1", fun () -> Assert.isTrue(1+1=2))
        Test("2+2", fun () -> Assert.isTrue(2+2=4))
    ]
    TestFolder("Arithmetic", [], tests)

let multiplications =
    let data = [ TestCase("1*1",  (1, 1, 1)); TestCase("2*2", (2, 2, 4)) ]
    let test(a:int, b:int, c:int) =
        Assert.isTrue(a * b = c)
    TestFolder.FromData("Multiplications", data, test)

let arithmetic =
    TestFolder("Arithmetic", [ additions; multiplications ], [])

// the entry point:

TestRunner.Load arithmetic

I don't know if it is possible to implement this using the existing standard dotnet test runners (https://github.com/microsoft/vstest etc.), or whether a completely new system would need to be created.

[baronfel]

@charlesroddie you're basically describing Expecto. It supports tests as first class values, and has a dotnet test adapter to allow for test running in a unified way. Basically all of my projects have been using it for years specifically for the organizational benefits you outline.

[charlesroddie]

Thanks for the suggestion @baronfel . I am exploring YoloDev.Expecto.Sdk but so far this doesn't allow a good connection to test explorer (folder hierarchy does not match up) haf/expecto#431 .

[dsyme]

@charlesroddie I'm labelling this as "probably not" because it's just very difficult to get into the test-framework level when things like Expecto, NUnit, XUnit exist, with all their pros and cons. An in-language solution would effectively compete with one of these but with some of the pros and some of the cons.

[charlesroddie]

-> @smoothdeveloper

[cartermp]

Yeah if this were all from scratch I'd actually love a Go-style approach where the convention is just filename_test, it lives side-by-side, and the testing tool just picks up those files by convention and works