Pin a precise version of Flutter

[gnprice]

So far, in the prototype phase, we've been able to just always use the latest Flutter version in a given channel (either beta or main). At some point we will want more control, specifying a particular version and being sure everyone gets that version even if it's not the very newest. That way everyone's seeing the same behavior, and a change happening outside our own repo can't break our tests.

In particular, if we use versions from the Flutter main branch, then that will be a constantly moving target. Using the moving target will still be OK as long as we don't frequently see anything change that breaks us, but at some point we'll want to pin the exact version to be sure of that.

Implementation notes

The pubspec.yaml file lets us specify a lower bound for the Flutter version, but not an upper bound. (The syntax permits an upper bound, but it's ignored.) In particular it doesn't permit specifying an exact version. Upstream discussions:

• [flutter_tools] pubspec.yaml environment.flutter constraint not respected by pub get flutter/flutter#95472 (comment)
• Flutter Wrapper Support. flutter/flutter#26577 (comment)
• It should be possible to pin Flutter Projects to a specific Flutter version flutter/flutter#40428
• dependencies change for flutter depending on channel/minor update? (material_color_utilities added in dev but not stable) flutter/flutter#95807 (comment)

Relatedly, while the flutter tool provides flutter upgrade as a handy way to get the latest Flutter version within any given channel (stable, beta, or main), there's no such way to get a specific version. So if Flutter did respect the pubspec.yaml file as far as complaining when the version was newer than specified, it still wouldn't provide a convenient way to actually get the specified version.

From those upstream threads, it seems like the usual solution in the Flutter community is to use one of several tools people have built to manage Flutter versions. I haven't looked into these in detail yet at all, but listing the ones I've seen mentioned:

• https://fvm.app/
• https://github.com/passsy/flutter_wrapper

So when pursuing this, we'll want to look at those and also look to see if there are other implementations to consider.

[gnprice]

This caused a bit of confusion in a recent PR:

• msglist: add scroll-to-bottom button to MessageList #223 (comment)

We also had an outright build-break a couple of months ago:

• 2f0f469 (that fix was from Parameterize the ApiConnection in getServerSettings. #103)

And there's the recurring small nuisance of:

• pubspec.lock sometimes doesn't work as a lockfile #85

So while our README says that our lack of pinning so far hasn't been a problem, that seems to no longer be true. It's not an overwhelming problem, but it is an actual problem.

---

Also, though this next part hasn't yet concretely bitten us, I'm concerned that it will get increasingly hard to run old versions of the app — say, to test whether some behavior is a regression or bisect when it changed — because there's no automated way to get a corresponding Flutter version, and no way at all to determine the exact Flutter versions that we were running at the time we were developing a given version of the app.

And unlike many kinds of problems, that isn't one where it necessarily makes sense to wait until we hit the problem before acting to solve it. That's because when you hit this problem, what you want isn't to have a solution now, but to have had a solution back when the old versions in question were developed, perhaps months earlier.

---

So I'm bumping this up in the timeline. I don't expect to have time to pick it up immediately, though.

[gnprice]

Here's a couple more thoughts about our needs here which I think aren't yet explicit above. (These are copied from a comment I just made at #577 (comment) , on a draft PR toward an implementation of this.)

• I'd really like the pinned version to apply when building or running the app — so flutter build, flutter run, and when hitting the "run" button or equivalent in an IDE — and not only when running tests.

  Similarly, it's common to use flutter test directly (or a "run this test" button in an IDE) rather than tools/check, because that gives control over which specific tests to run. If that uses a different version of Flutter than tools/check does, that's likely to be super confusing when one way doesn't reproduce the same failures as the other way.

  Accomplishing that may require some sort of setup step, which is OK (though I'll be interested in keeping that setup as clean as possible). The key is just that when someone clones the zulip-flutter repo and follows our setup instructions in the most straightforward way, they consistently get our pinned Flutter version across all the above ways of building or running our code.

• It's pretty common for us to develop changes to contribute to Flutter upstream, and that's been quite valuable for us. That's currently an extremely low-friction thing to do: the flutter tool is already using your local Git clone of the Flutter repo, so you just start editing that — and committing, and using git reset to move to various branches, etc. — and the changes show up just as seamlessly as if you were making them in your zulip-flutter tree itself.

  So I'd quite like to preserve that property, or close to it. Given the nature of pinning, we'll probably need to introduce a nonzero amount of friction — you're not really using a pinned version once you start making edits — but I'd like to keep that friction to the absolute minimum possible. For example, having to do something simple to disable pinning (so that you can make changes) would be fine.

[gnprice]

Further design discussion from #577:

• I'm not thrilled with fvm, having now looked closer at it.
• I sketched a possible approach for managing the pinned Flutter tree with a small script of our own.

[gnprice]

@chrisirhc also wrote:

Thinking on this PR more, I think there's some value with testing the commit with the latest commit on flutter. Something like check-flutter-nightly job. This helps with continuously verifying that zulip is compatible with newer versions of flutter and would allow you to skip testing it and just upgrade to a new (tested) version.
However, I saw that this behaviour is not supported on GitHub actions: actions/runner#2347 .
So I'm going to leave that check-flutter-nightly idea out for now.

I agree, a job like that would be valuable (and that's something we currently get from our unpinned setup). It's a shame that GitHub Actions lacks the feature to support that well (by having a job where failure doesn't turn the whole commit's build status to red). It looks like this is the main thread now for that feature request:
https://github.com/orgs/community/discussions/15452
but it's not particularly more promising. (A PM showed up in the thread a bit over a year ago, had some offline conversations, but never followed up in the thread.)

One possible workaround would be to have the job but only run it on main, not on PRs. That way when things started failing because of an upstream change, it wouldn't interfere with seeing the status of PRs — but we would still get the signal that something's wrong, as a prompt for us to investigate and fix.