dhall-docs test failures

[felixonmars]

I am getting the following test failures for dhall-docs 1.0.5.

I have set memory-0.16.0 as allow-newer, though. Not sure if it has anything to do with this...

Running 2 test suites...
Test suite doctest: RUNNING...
Test suite doctest: PASS
Test suite logged to: dist/test/dhall-docs-1.0.5-doctest.log
Test suite tasty: RUNNING...
dhall-docs
  doc-generation
    ImportAsType.dhall.html:                                                    FAIL
      Result did not match golden value.
    InvalidBlockComment.dhall.html:                                             FAIL
      Result did not match golden value.
    InvalidConsecutiveComments.dhall.html:                                      FAIL
      Result did not match golden value.
    InvalidMarkdown.dhall.html:                                                 FAIL
      Result did not match golden value.
    JumpToDefOnUnused.dhall.html:                                               FAIL
      Result did not match golden value.
    JumpToHereImports.dhall.html:                                               FAIL
      Result did not match golden value.
    JumpToLamBindingComplex.dhall.html:                                         FAIL
      Result did not match golden value.
    JumpToLamBindingSimple.dhall.html:                                          FAIL
      Result did not match golden value.
    JumpToLamBindingWithIndex.dhall.html:                                       FAIL
      Result did not match golden value.
    JumpToLamBindingWithQuotes.dhall.html:                                      FAIL
      Result did not match golden value.
    JumpToLamBindingWithShadowing.dhall.html:                                   FAIL
      Result did not match golden value.
    JumpToLetBindingSimple.dhall.html:                                          FAIL
      Result did not match golden value.
    JumpToLetBindingWithIndex.dhall.html:                                       FAIL
      Result did not match golden value.
    JumpToLetBindingWithQuotes.dhall.html:                                      FAIL
      Result did not match golden value.
    JumpToLetBindingWithShadowing.dhall.html:                                   FAIL
      Result did not match golden value.
    JumpToLetbindingComplex.dhall.html:                                         FAIL
      Result did not match golden value.
    JumpToRecordFieldWhenFieldDoesNotExist.dhall.html:                          FAIL
      Result did not match golden value.
    JumpToRecordFieldWhenLetAnnotationPresentShouldIgnoreAnnotation.dhall.html: FAIL
      Result did not match golden value.
    JumpToRecordFieldWhenVarIsAnnotatedWithRecordType.dhall.html:               FAIL
      Result did not match golden value.
    JumpToRecordFieldWhenVarIsOfRecordType.dhall.html:                          FAIL
      Result did not match golden value.
    JumpToRecordFieldWhenVarIsOfRecordTypeComplex.dhall.html:                   FAIL
      Result did not match golden value.
    JumpToRecordFieldWhenVarIsOfRecordTypeDeep.dhall.html:                      FAIL
      Result did not match golden value.
    JumpToRecordFieldWhenVarIsOfRecordTypeFromLamBinding.dhall.html:            FAIL
      Result did not match golden value.
    JumpToRecordFieldWhenVarIsOfRecordTypeOnPunnedEntry.dhall.html:             FAIL
      Result did not match golden value.
    JumpToRecordFieldWhenVarIsOfRecordTypeTransitivity.dhall.html:              FAIL
      Result did not match golden value.
    JumpToRecordFieldWhenVarIsOfRecordTypeWithDotSyntax.dhall.html:             FAIL
      Result did not match golden value.
    JumpToSelf.dhall.html:                                                      FAIL
      Result did not match golden value.
    JumpToUrls.dhall.html:                                                      FAIL
      Result did not match golden value.
    MarkdownExample.dhall.html:                                                 FAIL
      Result did not match golden value.
    MultilineIndentationExample.dhall.html:                                     FAIL
      Result did not match golden value.
    NoDoc.dhall.html:                                                           FAIL
      Result did not match golden value.
    NonDhallDocsCommentAfterValid.dhall.html:                                   FAIL
      Result did not match golden value.
    OrdinaryAnnotation.dhall.html:                                              FAIL
      Result did not match golden value.
    Pair.dhall.html:                                                            FAIL
      Result did not match golden value.
    RenderTypeIndexesExample.dhall.html:                                        FAIL
      Result did not match golden value.
    TwoAnnotations.dhall.html:                                                  FAIL
      Result did not match golden value.
    a/JumpToParent.dhall.html:                                                  FAIL
      Result did not match golden value.
    a/b.dhall.html:                                                             FAIL
      Result did not match golden value.
    a/b/JumpToGrandfather.dhall.html:                                           FAIL
      Result did not match golden value.
    a/b/c.dhall.html:                                                           FAIL
      Result did not match golden value.
    a/b/c/a.dhall.html:                                                         FAIL
      Result did not match golden value.
    a/b/c/index.html:                                                           FAIL
      Result did not match golden value.
    a/b/index.html:                                                             FAIL
      Result did not match golden value.
    a/index.html:                                                               FAIL
      Result did not match golden value.
    deep/nested/folder/index.html:                                              FAIL
      Result did not match golden value.
    deep/nested/folder/my-even.dhall.html:                                      FAIL
      Result did not match golden value.
    deep/nested/index.html:                                                     FAIL
      Result did not match golden value.
    index.html:                                                                 FAIL
      Result did not match golden value.
    package.dhall.html:                                                         FAIL
      Result did not match golden value.
  comments
    ignored comments
      discover
        ./tasty/data/comments/empty/SeveralLines:                               OK
        ./tasty/data/comments/empty/SingleLine:                                 OK
        ./tasty/data/comments/empty/Empty:                                      OK
        ./tasty/data/comments/empty/SingleBlock:                                OK
        ./tasty/data/comments/empty/Mixed:                                      OK
    invalid comments
      discover
        ./tasty/data/comments/invalid/MissingNewLineBlock:                      OK
        ./tasty/data/comments/invalid/SingleLinesBadAligned:                    OK
        ./tasty/data/comments/invalid/TwoValidMixed:                            OK
        ./tasty/data/comments/invalid/TwoValidSingleLines:                      OK
        ./tasty/data/comments/invalid/TwoValidBlocks:                           OK
    valid comments
      discover
        tasty/data/comments/valid/BlockWithOtherComments:                       OK
        tasty/data/comments/valid/SingleLinesEmptyLine:                         OK
        tasty/data/comments/valid/SingleLInesWithIndentation:                   OK
        tasty/data/comments/valid/SingleLineWithOtherComments:                  OK
        tasty/data/comments/valid/BlockWithIndentation:                         OK
        tasty/data/comments/valid/SingleLineWithPrefixWhitespace:               OK
        tasty/data/comments/valid/SingleLinesIgnoreLInesBeforeMarker:           OK
        tasty/data/comments/valid/SingleLinesEmptyLineWithoutComment:           OK
        tasty/data/comments/valid/SingleLines:                                  OK
        tasty/data/comments/valid/Block:                                        OK
        tasty/data/comments/valid/BlockWithWhitespaceLines:                     OK
        tasty/data/comments/valid/BlockWithEmptyLinesIgnored:                   OK

49 out of 71 tests failed (0.01s)
Test suite tasty: FAIL
Test suite logged to: dist/test/dhall-docs-1.0.5-tasty.log
1 of 2 test suites (1 of 2 test cases) passed.

[sjakobi]

I have also encountered these test failures while working on the support for GHC 9.0.1 (#2154). AFAIK they are due to changes in the Hashable Text instance that affect the output from lucid (https://github.com/chrisdone/lucid/issues/107).

I think we'll need to constrain the hashable version used in the dhall-docs tests – #2154 contains a patch for doing that.

[felixonmars]

Thanks for the info. I am currently ignoring the errors then. Hopefully there will be a way to compare the result without depending on the specific ordering.

[sjakobi]

This was fixed in #2319.