Commits on Mar 6, 2020

1. Add test run that uses www feature flags …

   In CI, we run our test suite against multiple build configurations. For
   example, we run our tests in both dev and prod, and in both the
   experimental and stable release channels. This is to prevent accidental
   deviations in behavior between the different builds. If there's an
   intentional deviation in behavior, the test author must account
   for them.
   
   However, we currently don't run tests against the www builds. That's
   a problem, because it's common for features to land in www before they
   land anywhere else, including the experimental release channel.
   Typically we do this so we can gradually roll out the feature behind
   a flag before deciding to enable it.
   
   The way we test those features today is by mutating the
   `shared/ReactFeatureFlags` module. There are a few downsides to this
   approach, though. The flag is only overridden for the specific tests or
   test suites where you apply the override. But usually what you want is
   to run *all* tests with the flag enabled, to protect against unexpected
   regressions.
   
   Also, mutating the feature flags module only works when running the
   tests against source, not against the final build artifacts, because the
   ReactFeatureFlags module is inlined by the build script.
   
   Instead, we should run the test suite against the www configuration,
   just like we do for prod, experimental, and so on. I've added a new
   command, `yarn test-www`. It automatically runs in CI.
   
   Some of the www feature flags are dynamic; that is, they depend on
   a runtime condition (i.e. a GK). These flags are imported from an
   external module that lives in www. Those flags will be enabled for some
   clients and disabled for others, so we should run the tests against
   *both* modes.
   
   So I've added a new global `__VARIANT__`, and a new test command `yarn
   test-www-variant`. `__VARIANT__` is set to false by default; when
   running `test-www-variant`, it's set to true.
   
   If we were going for *really* comprehensive coverage, we would run the
   tests against every possible configuration of feature flags: 2 ^
   numberOfFlags total combinations. That's not practical, though, so
   instead we only run against two combinations: once with `__VARIANT__`
   set to `true`, and once with it set to `false`. We generally assume that
   flags can be toggled independently, so in practice this should
   be enough.
   
   You can also refer to `__VARIANT__` in tests to detect which mode you're
   running in. Or, you can import `shared/ReactFeatureFlags` and read the
   specific flag you can about. However, we should stop mutating that
   module going forward. Treat it as read-only.
   
   In this commit, I have only setup the www tests to run against source.
   I'll leave running against build for a follow up.
   
   Many of our tests currently assume they run only in the default
   configuration, and break when certain flags are toggled. Rather than fix
   these all up front, I've hard-coded the relevant flags to the default
   values. We can incrementally migrate those tests later.

   

   acdlite committed Mar 6, 2020
   Configuration menu

   • View commit details
   • Copy full SHA for 89ba911
   • Browse repository at this point

   Copy the full SHA

   89ba911 View commit details

   Browse the repository at this point in the history