Reduce use of generators in finding files to document

[srawlins]

The short story of this change is that we're refactoring the "file-finding" code so that _findFilesToDocumentInPackage is no longer async*, and _listDir is no longer sync*. Lazy generators like these are nice when the full sequence of elements may not be generated. However, in this code, we definitely iterate over every element returned by these two functions, and sync* is known to be slow, and a package set may be thousands of files.

The longer story has more details:

• findFilesToDocumentInPackage was async because of the code that finds a package config, and that code is only needed when the includeDependencies parameter is true. The function only has two callers, one of which always passed includeDependencies: false.
  • So we can extract out the code that looks for a package config, and iterates over the listed packages, into a new function, _findPackagesToDocument, only called by _getFilesToDocument. Now _findFilesToDocumentInPackage doesn't need to be async, or a generator.
• Further efficiencies are found in getFilesToDocument: This function can actually be split into two cases: are we documenting the SDK, or not?
  • If documenting the SDK, we don't need to consider the auto-include-dependencies flag, as it is never used when documenting the SDK. We also don't need to consider the deprecated include-externals flag, as it is also never used when documenting the SDK. So we return a very simple calculation in this case. When documenting the SDK, the code is now simpler and does less work.
  • We can then isolate all of the complexity in the "not documenting the SDK" case, taking into account these two flags, and the embedder SDK files.
• Now that _discoverLibraries is only called once, we don't need to pass in the addLibrary callback. Instead, _discoverLibraries can directly call uninitializedPackageGraph.addLibraryToGraph.

I benchmarked this both documenting the SDK and googleapis package. There was no visible difference in time-to-document or memory usage when documenting the SDK. For the googleapis package, there was a -1% time-to-document, and -0.5% memory usage, which might well be within the error margins. 🤷

---

• I’ve reviewed the contributor guide and applied the relevant portions to this PR.

Contribution guidelines:

• See our contributor guide for general expectations for PRs.
• Larger or significant changes should be discussed in an issue before creating a PR.
• Contributions to our repos should follow the Dart style guide and use dart format.
• Most changes should add an entry to the changelog and may need to rev the pubspec package version.
• Changes to packages require corresponding tests.

Note that many Dart repos have a weekly cadence for reviewing PRs - please allow for some latency before initial review feedback.

[srawlins]

CC @dart-lang/analyzer-team

@bwilkerson I was looking at this code while reminding myself how dartdoc looks for files to document. I didn't get far before I decided to simplify this code. :/

[srawlins]

This is still ready for review.