Document the process and get things ready

.cirrus.yml

@@ -0,0 +1,21 @@
+task:
+  name: tests-linux
+  use_compute_credits: $CIRRUS_USER_COLLABORATOR == 'true' && $CIRRUS_PR == ''
+  script: scripts/verify_tests_on_master.sh
+  container:
+    image: gcc:latest
+
+task:
+  name: tests-windows
+  use_compute_credits: $CIRRUS_USER_COLLABORATOR == 'true' && $CIRRUS_PR == ''
+  script: scripts\verify_tests_on_master.bat
+  windows_container:
+    image: cirrusci/windowsservercore:2019
+    os_version: 2019
+
+task:
+  name: tests-macos
+  use_compute_credits: $CIRRUS_USER_COLLABORATOR == 'true' && $CIRRUS_PR == ''
+  script: scripts/verify_tests_on_master.sh
+  osx_instance:
+    image: mojave-base

.gitignore

@@ -0,0 +1 @@
+flutter

README.md

@@ -1,2 +1,110 @@
-# tests
-Contributed tests for Flutter
+# Flutter Tests
+
+This repository contains references to tests that are run with every
+commit to Flutter to verify that no breaking changes have been
+introduced. The tests referenced by this repository are typically
+maintained by people outside of the Flutter team, as part of the
+development of their applications. They are intended to give the
+Flutter team visibility into how their changes affect real-world
+developers using Flutter.
+
+## Adding more tests
+
+You are welcome to add a new batch of tests. To do so, copy the file
+`registry/template.test` to create a new file in the `registry/`
+directory. Fill in the fields and delete all the comments. Then,
+submit a PR with this new file.
+
+Tests must fulfill the following criteria to be added:
+
+* All the code must be available publicly on GitHub under a license
+  compatible with this effort.
+
+* Tests must be hermetic. For example, a test should not involve
+  network activity, spawn processes, or access the local file system
+  except to access files that are packaged with the test.
+
+* Tests must be resilient to being run concurrently with other tests,
+  including concurrent runs of themselves.
+
+* Tests must be reliable. A test must not claim to pass if it is
+  failing. Running a test multiple times in a row must always have the
+  same result.
+
+* Tests must have no output when they are passing.
+
+* Tests must be as fast as possible given the hardware. For example,
+  tests must not use real timers or depend on the wall clock.
+
+* The time taken by tests must be proportional to their value. A few
+  thousands tests are expected to run within a few minutes. An upper
+  limit of about five minutes will be applied to each contributed test
+  suite (not including the time to download the tests), but it is
+  expected that most suites will complete in seconds.
+
+* The tests must be compatible with any tools for automatically
+  updating Flutter code (e.g. they cannot rely on custom code
+  generation unless such code generation can hook into the automatic
+  update mechanism).
+
+* The tests must represent good practices as described in Flutter's
+  documentation. For example, using an object after calling its
+  `dispose` method violates the contract described by that method.
+  Accessing the fields of a private `State` subclass from another
+  package by casting it to dynamic is similarly sketchy and would not
+  be supported behaviour.
+
+* The tests must pass at the time they are contributed.
+
+
+## Running the tests locally
+
+To run these tests locally, check out this directory in a directory
+parallel to your `flutter` repository checkout, then, from this
+directory, run:
+
+```
+../flutter/bin/cache/dart-sdk/bin/dart ../flutter/dev/customer_testing/run_tests.dart --skip-template --verbose registry/*.test
+```
+
+
+## If a test is broken
+
+The point of these tests is to make sure we don't break existing code,
+while still being able to make improvements to Flutter APIs.
+
+If you find that a PR you have created in flutter/flutter causes one
+these tests to fail, you have the following options:
+
+1. Change your PR so that the test no longer fails. This is the
+   preferred option, so long as the result is one we can be proud of.
+   Is the resulting API something that you would plausibly come up
+   with without the backwards-compatibility constraint? That's good.
+   Is the resulting API something that, as soon as you see it, you
+   think "why?" or "that's weird"? That's bad. Consider the advice in
+   the Style guide:
+   https://github.com/flutter/flutter/wiki/Style-guide-for-Flutter-repo
+
+2. Go through the breaking change process, as documented here:
+   https://github.com/flutter/flutter/wiki/Tree-hygiene#handling-breaking-changes
+   If you're going to do this, you will need to contact the relevant
+   people responsible for the breaking test(s) (see the relevant .test
+   files), help them fix their code, and update this repository to use
+   the new version of their tests, in addition to the steps described
+   on the wiki.
+
+3. Remove the test in question. This is by far the least ideal
+   solution. To go down this path, we must first establish that one of
+   the following is true:
+
+    - the people listed as contacts for the test are not responsive.
+
+    - the test is poorly written (e.g. it contains a race condition or
+      relies on assumptions that violate clearly documented API
+      contracts), and the people listed as contacts are not willing to
+      fix the test or accept fixes for the test.
+
+    - we have gone through the breaking change process cited above,
+      but are unable to update the test accordingly (e.g. the people
+      listed as contacts are not willing to work with us to update
+      their code).

registry/DanTup_tiler.test

@@ -0,0 +1,7 @@
+contact=danny@tuppeny.com
+fetch=git clone https://github.com/DanTup/tiler.git tests
+fetch=git -C tests checkout a448ce4781f02d11fbf9c310502a6b4382809ecc
+fetch=git -C tests submodule init
+fetch=git -C tests submodule update --recursive --remote
+update=.
+test=flutter test test

registry/flutter_svg.test

@@ -0,0 +1,5 @@
+contact=dfield@gmail.com
+fetch=git clone https://github.com/dnfield/flutter_svg.git tests
+fetch=git -C tests checkout a091088418dcb2acbbd103b0472f9dc309bfda2f
+update=.
+test=flutter test

registry/template.test

@@ -0,0 +1,107 @@
+# This is a template for adding tests to the flutter/tests registry.
+#
+# To submit a new batch of tests, copy this file to create a new file
+# in this directory, update it as described below, and submit it as a
+# new PR to this repository.
+#
+# New files should be named `foo.test` where `foo` is a description of
+# the project whose tests are being added. (The file named
+# `template.test` is skipped automatically.)
+#
+# Lines beginning with a "#" are ignored. Delete all such lines from
+# this template before submitting a new `.test` file (you may add new
+# comments if you have something to say that doesn't fit in one of the
+# meaningful lines). Also delete blank lines like the following:
+
+# The contact lines must contain the e-mail addresses, one to a line,
+# of one or more people who will be able to help if a problem is found
+# with a test. People on this list should be able to respond to
+# e-mails within about 72 hours. If there is a problem with a test and
+# we cannot reach any of the people listed, then we will probably
+# remove the tests.
+contact=foo@example.com
+contact=bar@example.com
+
+# The fetch line(s) must contain one or more commands that fetch the
+# tests into a new directory called "tests" in the current directory.
+# The example below clones the dnfield/flutter_svg GitHub repository
+# then checks out a specific revision.
+#
+# Our current policy is that all tests should be in public GitHub
+# repositories. (This way, we don't take on additional dependencies on
+# other services that may or not go down. We already depend on GitHub,
+# so if GitHub goes down it doesn't matter that we can't reach
+# contributed tests.) If you have some tests to contribute that are
+# not and will not be in GitHub, please contact ian@hixie.ch before
+# submitting a PR proposing to add such tests. We may make exceptions
+# if your tests are especially valuable.
+#
+# The checkout must always check out the exact same set of tests. Do
+# not check out the current master branch or some such; always fetch a
+# specific revision. You may regularly submit PRs that update the
+# revision being used.
+#
+# Each fetch directive is processed as follows: the value is split on
+# spaces (without honoring quotes or backslash escapes or anything
+# like that), then the first segment is used as the executable to ask
+# the operating system to run, and the rest are treated as arguments
+# to pass to that process.
+#
+# The commands specified on the "fetch" lines must return the exit
+# code 0 or else the tests will be skipped entirely. The "git" program
+# will be on the path.
+#
+# There must be at least one "fetch" line. (In fact currently there
+# must be at least two, and the first two must follow the pattern
+# below more or less exactly. Additional lines are allowed e.g. to
+# initialize and update submodules, but this should be rare. If this
+# is a problem, let us know in your PR.)
+fetch=git clone https://github.com/USERNAME/REPOSITORY.git tests
+fetch=git -C tests checkout 0123456789abcdef0123456789abcdef01234567
+
+# The "update" lines specify paths (relative to the newly created
+# "tests" directory) in which to run a flutter command to
+# automatically update the code to fix breaking changes. The command
+# will be run in turn in each specified directory. The root can be
+# specified by using the path "." as that represents the "current"
+# directory. Multiple paths may be specified by specifying multiple
+# "update" lines.
+#
+# The command that updates the code may fail, in which case the tests
+# are considered to have failed. This helps catch cases where
+# automatically updating the code doesn't work.
+#
+# There must be at least one "update" line.
+update=.
+
+# The test lines specify commands to run to execute the actual tests.
+# Each of these should return either 0 to indicate success, or a
+# non-zero exit code to report failure.
+#
+# The "flutter" tool and the "dart" program will be in the path.
+#
+# If there are multiple directories, create a shell script in the
+# repository to enter each directory and run the tests in that
+# directory, returning non-zero as soon as any set of tests fails. You
+# can have multiple such scripts. In the example below, we run
+# "flutter test" in the root of the repository, a "more_tests.sh"
+# shell script from that same directory (the "./" indicates it isn't a
+# command to search for on the path), and the "extra_tests.dart"
+# program in the "dev" directory, run via "dart".
+test=flutter test
+test=./more_tests.sh
+test=dart dev/extra_tests.dart
+
+# To test your tests, check out the flutter/flutter repository and
+# then, from the root of that repository, run:
+#   bin/flutter; time bin/cache/dart-sdk/bin/dart dev/customer_testing/run_tests.dart --repeat=100 <path>
+# ...where <path> is the path to this file. This should run with no
+# output, without failing, and should in total take less than 15
+# minutes (a hundred runs of your tests taking just a few seconds
+# each). If your tests take longer, mention this in your PR and we
+# will evaluate them to see how valuable they are (e.g. how unique and
+# different they are compared to other tests we already have). The
+# more valuable the tests, the more likely we are to accept them
+# despite them taking a long time to run.
+
+# See also the `README.md` file in the root of this repository.

scripts/verify_tests_on_master.bat

@@ -0,0 +1,23 @@
+@ECHO ON
+
+REM Fetch Flutter.
+git clone https://github.com/flutter/flutter.git || GOTO :END
+CALL flutter\bin\flutter doctor -v || GOTO :END
+
+@ECHO.
+@ECHO.
+
+REM Now run the tests a bunch of times to try to find flakes (tests that sometimes pass
+REM even though they should be failing).
+CALL flutter\bin\cache\dart-sdk\bin\dart flutter\dev\customer_testing\run_tests.dart --repeat=15 --skip-template registry/*.test || GOTO :END
+
+@ECHO.
+@ECHO.
+@ECHO Testing complete.
+GOTO :EOF
+
+:END
+@ECHO.
+@ECHO.
+@ECHO Testing failed.
+EXIT /B 1

scripts/verify_tests_on_master.sh

@@ -0,0 +1,9 @@
+set -ex
+
+# Fetch Flutter.
+git clone https://github.com/flutter/flutter.git
+flutter/bin/flutter doctor -v
+
+# Now run the tests a bunch of times to try to find flakes (tests that sometimes pass
+# even though they should be failing).
+flutter/bin/cache/dart-sdk/bin/dart flutter/dev/customer_testing/run_tests.dart --repeat=15 --skip-template registry/*.test