[red-knot]: Add a VendoredFileSystem implementation

[AlexWaygood]

Summary

Work towards #11653.

This PR adds a VendoredFileSystem implementation to the ruff_db crate, the idea being that this is the filesystem we would use for resolving vendored typeshed stubs.

Some of this feels a bit awkward, but I'm not sure there's a better way. I ran into two problems while writing this PR:

1. The ZipArchive struct exposed by the zip crate takes &mut self for most of the methods that query the underlying archive for information about files or directories within the archive. This means that in order to query zipped data from trait method implementations that take &self, we end up having to clone the underlying archive. I think that's okay, as the zip docs say that it should be cheap to clone (at least right now), but it didn't fill me with joy.
2. The camino crate strips trailing slashes in a lot of its methods, as they're not semantically significant for most paths. However, for zip archives, trailing slashes are very significant: if a stdlib/ directory exists inside the zip, zip.by_name("stdlib") will not resolve to that directory (only zip.by_name("stdlib/") will). I've attempted to abstract over this difference to other kinds of paths, so that the VendoredFileSystem is consistent with the other FileSystem implementations in this respect. But it was slightly tricky to do so.

Test Plan

I added lots of parameterized tests.

[github-actions]

ruff-ecosystem results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

Formatter (stable)

✅ ecosystem check detected no format changes.

Formatter (preview)

✅ ecosystem check detected no format changes.

[MichaReiser]

Nice work.

Overall looks good, but we should address the following items before merging:

• Use regular functions for tests instead of macros
• Remove FileSystem implementation and use VendoredFilePath and VendoredFilePathBuf (and move the definitions into `vendored.rs)
• Move the implementation out of the vendored directory.
• If we can, remove the probing where we append a slash at the end of the path when reading failed. This seems like trial and error.

One open question for me is whether we should create a dedicated ruff_vendored crate that also contains all vendored files (and contains the build.rs). The benefit of that would be that integration tests can use the actual vendored files, which I think would be desired (but use stubed vendored files for unit tests). If that's undesired, I think leaving it in ruff_db is fine.

[MichaReiser]

I think we could combine this with the VendoredFilePath and VendoredFilePathBuf where we require the use of VendoredFilePathBuf::new (that normalizes the string).

[AlexWaygood]

• Move the implementation out of the vendored directory.

I'm guessing you mean file_system directory? :)

[AlexWaygood]

One open question for me is whether we should create a dedicated ruff_vendored crate that also contains all vendored files (and contains the build.rs). The benefit of that would be that integration tests can use the actual vendored files, which I think would be desired (but use stubed vendored files for unit tests). If that's undesired, I think leaving it in ruff_db is fine.

I wasn't sure while writing this the extent to which we'd want to tie the VendoredFileSystem implementation to the exact details of the only repository we realistically expect to be vendoring (typeshed). In this implementation, I tried to keep things relatively abstract, so that it could theoretically work with any vendored zip archive. That means that I expect to have to build another layer of abstraction on top of this for working with typeshed specifically, because we have to take the VERSIONS file into account when deciding whether a file "exists" in typeshed. If a file stdlib/foo.pyi exists in the typeshed zip archive but VERSIONS tells us that it hadn't actually been added to the stdlib yet on the Python version the user has selected to be the target version, then the query to our typeshed abstraction should arguably report that the stdlib/foo.pyi doesn't exist at all. That's the reality at runtime on that Python version, after all: if the module hasn't been added to the stdlib yet, the file for that module physically doesn't exist at runtime on that Python version.

[MichaReiser]

This overall looks good but I think we can make it much simpler by removing all code that's only necessary for handling directories. From what I understand, we won't need directory support to implement module resolution. All we need is to lookup a specific file path. Or are there operations that need to know about the existence of a directory?

[MichaReiser]

Nit: I would remove the inner file type. It doesn't provide that much functionality but the nesting becomes somewhat hard to understand. It also allows to remove other wrapper types, reducing the overall code.

[AlexWaygood]

I got rid of one of the other wrapper types in f4f97d4, but I think I would prefer to keep using an inner file type for this. I tried changing it to a type alias, and in my opinion it made the VendoredFileSystem unduly coupled to the inner representation that used a lock/RefCell; I also found it less readable.

[MichaReiser]

Should lookup_path take a &VendoredPath instead and do all the normalization?

[AlexWaygood]

That's a nice idea.

[AlexWaygood]

Because of the "trailing slashes are semantically significant for zip archives" thing combined with the "ZipFile::by_name() requires a mutable reference" thing, it's hard for me to see how we could do this while keeping the borrow checker happy and avoiding doing path normalization twice. It doesn't seem possible to call self.0.by_name() twice in lookup_path() while keeping the borrow checker happy, unfortunately (unless I'm missing something), and because of the trailing-slash thing, we need to call it twice in lookup_path() if we're going to do the normalization inside lookup_path().

[MichaReiser]

Nit: Maybe just self.metadata(path).is_some()

[AlexWaygood]

I couldn't resist micro-optimising this 😄 (this avoids calling zip_file.crc32() and zip_file.is_dir(), which aren't necessary if you only need to know whether the path exists or not)