fix: golden tests don't work with test optimisation flag

[wolfenrain]

Description

Testing goldens don't work when using the test optimisation. It throws an error, only work around at the moment is not using optimisation.

Steps To Reproduce

1. Write a test with a golden in it.
2. Run very_good test --update-goldens
3. Run very_good test --optimization
4. See error

Expected Behavior
That I can run my golden tests while utilising test optimisation.

Additional Context

Running "flutter test" in /Users/wolfen/Projects/some_project...
✓ Optimizing tests (5.0s)
══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
The following TestFailure was thrown running a test:
Expected: one widget whose rasterized image matches golden image "golden/my_component.png"
  Actual: _WidgetPredicateFinder:<exactly one widget with widget matching predicate (Closure:
(Widget) => bool) (ignoring offstage widgets): GameWidget<TestGame>(state:
_GameWidgetState<TestGame>#11074)>
   Which: Could not be compared against non-existent file: "golden/my_component.png"

When the exception was thrown, this was the stack:
#0      fail (package:test_api/src/expect/expect.dart:137:31)
#1      _expect.<anonymous closure> (package:test_api/src/expect/expect.dart:112:9)
<asynchronous suspension>
<asynchronous suspension>
(elided 6 frames from dart:async and package:stack_trace)

The test description was:
  renders correctly
════════════════════════════════════════════════════════════════════════════════════════════════════
Test failed. See exception logs above.
The test description was: renders correctly
components_my_component_test_dart MyComponent renders correctly /Users/wolfen/Projects/some_project/test/.test_runner.dart (FAILED)
00:04 +1 -1: Some tests failed.

[Giuspepe]

Hey @wolfenrain,

the problem is that the goldenFileComparator looks for the golden files relative from the file containing the golden test. So if you run a golden test from the file /Users/me/my_flutter_project/test/features/foo/foo_golden_test.dart which uses a something like matchesGoldenFile('golden/foo.png') or the goldenTest method from the Alchemist package, the goldenFileComparator will look for the file under /Users/me/my_flutter_project/test/features/foo/golden/foo.png.

Now when you use the test optimization, all tests are executed from Users/me/my_flutter_project/test/.test_runner.dart. Thus, the goldenFileComparator will search for the golden file relative from the .test_runner.dart file instead of relative from the original foo_golden_test.dart. So it wrongly expects the golden file to be at /Users/me/my_flutter_project/test/golden/foo.png instead of /Users/me/my_flutter_project/test/features/foo/golden/foo.png.

I found a pretty nifty workaround to fix this: when running with test optimization, change the implementation of goldenFileComparator so it knows about the implications of the optimization and returns the correct paths of the golden files. Concretely this means adding the following code to the generated .test_runner.dart:

import 'dart:io';

import 'package:flutter_test/flutter_test.dart';
import 'package:path/path.dart';

// ... imports of all tests ...

class _TestOptimizationAwareGoldenFileComparator extends LocalFileComparator {
  final List<String> goldenFiles;

  _TestOptimizationAwareGoldenFileComparator()
      : goldenFiles = Directory.fromUri(
                (goldenFileComparator as LocalFileComparator).basedir)
            .listSync(recursive: true, followLinks: true)
            .whereType<File>()
            .map((file) => file.path)
            .where((path) => path.endsWith('.png'))
            .toList(),
        super(_getTestFile());

  static Uri _getTestFile() {
    final basedir =
        (goldenFileComparator as LocalFileComparator).basedir.toString();
    return Uri.parse("$basedir/.test_runner.dart");
  }

  @override
  Uri getTestUri(Uri key, int? version) {
    final keyString = fromUri(key);
    return Uri.parse(goldenFiles
        .singleWhere((goldenFilePath) => goldenFilePath.endsWith(keyString)));
  }
}

void main(){
  goldenFileComparator = _TestOptimizationAwareGoldenFileComparator();
  // ... execution of all tests ..
}

[felangel]

Hey @wolfenrain,

the problem is that the goldenFileComparator looks for the golden files relative from the file containing the golden test. So if you run a golden test from the file /Users/me/my_flutter_project/test/features/foo/foo_golden_test.dart which uses a something like matchesGoldenFile('golden/foo.png') or the goldenTest method from the Alchemist package, the goldenFileComparator will look for the file under /Users/me/my_flutter_project/test/features/foo/golden/foo.png.

Now when you use the test optimization, all tests are executed from Users/me/my_flutter_project/test/.test_runner.dart. Thus, the goldenFileComparator will search for the golden file relative from the .test_runner.dart file instead of relative from the original foo_golden_test.dart. So it wrongly expects the golden file to be at /Users/me/my_flutter_project/test/golden/foo.png instead of /Users/me/my_flutter_project/test/features/foo/golden/foo.png.

I found a pretty nifty workaround to fix this: when running with test optimization, change the implementation of goldenFileComparator so it knows about the implications of the optimization and returns the correct paths of the golden files. Concretely this means adding the following code to the generated .test_runner.dart:

import 'dart:io';

import 'package:flutter_test/flutter_test.dart';
import 'package:path/path.dart';

// ... imports of all tests ...

class _TestOptimizationAwareGoldenFileComparator extends LocalFileComparator {
  final List<String> goldenFiles;

  _TestOptimizationAwareGoldenFileComparator()
      : goldenFiles = Directory.fromUri(
                (goldenFileComparator as LocalFileComparator).basedir)
            .listSync(recursive: true, followLinks: true)
            .whereType<File>()
            .map((file) => file.path)
            .where((path) => path.endsWith('.png'))
            .toList(),
        super(_getTestFile());

  static Uri _getTestFile() {
    final basedir =
        (goldenFileComparator as LocalFileComparator).basedir.toString();
    return Uri.parse("$basedir/.test_runner.dart");
  }

  @override
  Uri getTestUri(Uri key, int? version) {
    final keyString = fromUri(key);
    return Uri.parse(goldenFiles
        .singleWhere((goldenFilePath) => goldenFilePath.endsWith(keyString)));
  }
}

void main(){
  goldenFileComparator = _TestOptimizationAwareGoldenFileComparator();
  // ... execution of all tests ..
}

Thanks for taking the time to investigate and for sharing your solution! We'll try to address this soon 👍

[wolfenrain]

This got accidentally closed, reopening it as a few changes are still needed.