Update Windows trampolines for compatibility with Python 3.7 and earlier

[nahco314]

Summary

uv-trampoline uses Python's zipimport feature in a slightly hacky way to provide exe files, but the current method causes errors in Python 3.7 and earlier.

Specifically, uv-trampoline stores the Python executable path and other information as archive comments in the zip file. However, zipimport before Python 3.7 does not support archive comments (https://docs.python.org/3/library/zipimport.html#:~:text=Changed%20in%20version%203.8%3A%20Previously%2C%20ZIP%20archives%20with%20an%20archive%20comment%20were%20not%20supported ).

When I actually run the installed exe with Python 3.7 on Windows, I get the following puzzling error.

$ pycowsay
  File "C:\Users\nahco\.local\bin\pycowsay.exe", line 1
SyntaxError: Non-UTF-8 code starting with '\x83' in file C:\Users\nahco\.local\bin\pycowsay.exe on line 2, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details

This PR will support Python 3.7 and earlier by moving the path to Python, etc. (as well as the exe file content) before the zip portion.

Is this necessary?

Of course, Python 3.7 is EOL.
But it is still used in practice and as long as it is installable with uv, it is a good thing to support it. The change to support it is simple and will not change the cost of maintenance much.

Also, if this change is unacceptable, I think we should display an error indicating that the exe is not supported in Python 3.7 or earlier. In that case I am still willing to create a PR.

Test Plan

Simply install and run some tool using Python 3.7.
I added this for now because I don't really understand the uv code based testing method, but I will fix it if there are any problems.

[zanieb]

Thanks for contributing!

Unfortunately, I also worked on the trampoline today and this will conflict with #8637 – I presume that can be reconciled though.

I'm a +1 on this. We'll want to hear from @konstin as well.

cc @samypr100 if you're interested, I know you've done a fair bit of work on the trampolines.

[zanieb]

I am a bit confused since I thought the trampolines were written when 3.7 was not EOL — did we break support at some point?

[nahco314]

I'm not sure, but at the time the trampoline in posy was created, 3.7 must have been somewhat old, so it may simply have gone unnoticed.

[konstin]

I'm -1 on supporting Python 3.7. Python 3.7 was already EOL when uv was released, and users should upgrade asap rather than patching the support into tools. Otherwise the code looks good.

[T-256]

Previously raised issue: #2445.
For some works I'm stuck at Python 3.7. On Windows I have this problem that I cannot run apps/scripts directly after activating venv, or neither via uv run.
There is limited workaround: uv run python -m <MODULE> (only for those has same script name exposed as module name).

I can verify uv fully works on Python 3.7 on Windows and the only issue remained is that trampoline issue.

[samypr100]

-1 for 3.7 support.
I'm not sure if the added complexity is worth it given now that 3.8 is also effectively EOL, so we'd be supporting two EOL versions. In addition, it adds additional overhead to the developer when testing as they'd have to test 3.7 doesn't break the trampolines to @notatallshaw 's point in #7418 (comment)

[T-256]

That's correct both 3.7 and 3.8 EOL, but I think uv is still able to support them as in:

uv/crates/uv-python/src/discovery.rs

Lines 1767 to 1770 in 9fa4fea

	/// Check if the request is for a version supported by uv.
	///
	/// If not, an `Err` is returned with an explanatory message.
	pub(crate) fn check_supported(&self) -> Result<(), String> {

• Also, see Discussion: Support statement or policy for python #8255

[zanieb]

One problem here is that I need to be able to sniff executables to tell if they're trampolines. I feel like that's trivial with the magic number at the end but kind of annoying here? If we're inverting the format I guess I'd expect

|       `launcher.exe`        |
| :-------------------------: |
| `<b'U', b'V', b'U', b'V'>`  |
| `<len(path to python.exe)>` |
|   `<path to python.exe>`    |
|  `<zipped python script>`   |

right?

[nahco314]

It would be difficult to put the magic number of the UV in a trivial position, since the magic number of the exe must be placed at the beginning of the file and the EOCD at the end.
The inverted version is also difficult because the length of the exe file is non-trivial. (The length of the exe file is known in advance, but this may not be very useful since it can change from version to version.)

I think it is possible to insert the magic number in the last 2 bytes of the EOCD (where the length of the comment is stored). Whether this will work is implementation-dependent, but a normal zip implementation should work fine.

$ ./.venv/Scripts/black.exe --version
black, 23.3.0 (compiled: yes)
Python (CPython) 3.7.9

$ od -c ./.venv/Scripts/black.exe | tail -n 3
0125700 005 006  \0  \0  \0  \0 001  \0 001  \0   9  \0  \0  \0   9 001
0125720  \0  \0   U   V
0125724

[zanieb]

Yeah sorry, I forgot that there's the entire trampoline executable itself at the front.

(The length of the exe file is known in advance, but this may not be very useful since it can change from version to version.)

We could hard-code this, I guess. Or pad to a specific size.

I basically need to be able to inspect the trampoline, as in

uv/crates/uv-trampoline-builder/src/lib.rs

Lines 39 to 123 in 9cb7bcf

	#[derive(Debug)]
	pub struct Launcher {
	pub kind: LauncherKind,
	pub python_path: PathBuf,
	}
	impl Launcher {
	/// Read [`Launcher`] metadata from a trampoline executable file.
	///
	/// Returns `Ok(None)` if the file is not a trampoline executable.
	/// Returns `Err` if the file looks like a trampoline executable but is formatted incorrectly.
	///
	/// Expects the following metadata to be at the end of the file:
	///
	/// ```text
	/// - file path (no greater than 32KB)
	/// - file path length (u32)
	/// - magic number(4 bytes)
	/// ```
	///
	/// This should only be used on Windows, but should just return `Ok(None)` on other platforms.
	///
	/// This is an implementation of [`uv-trampoline::bounce::read_trampoline_metadata`] that
	/// returns errors instead of panicking. Unlike the utility there, we don't assume that the
	/// file we are reading is a trampoline.
	#[allow(clippy::cast_possible_wrap)]
	pub fn try_from_path(path: &Path) -> Result<Option<Self>, Error> {
	let mut file = File::open(path)?;
	// Read the magic number
	let Some(kind) = LauncherKind::try_from_file(&mut file)? else {
	return Ok(None);
	};
	// Seek to the start of the path length.
	let Ok(_) = file.seek(io::SeekFrom::End(
	-((MAGIC_NUMBER_SIZE + PATH_LENGTH_SIZE) as i64),
	)) else {
	return Err(Error::InvalidLauncher(
	"Unable to seek to the start of the path length".to_string(),
	));
	};
	// Read the path length
	let mut buffer = [0; PATH_LENGTH_SIZE];
	file.read_exact(&mut buffer)
	.map_err(|err| Error::InvalidLauncherRead("path length".to_string(), err))?;
	let path_length = {
	let raw_length = u32::from_le_bytes(buffer);
	if raw_length > MAX_PATH_LENGTH {
	return Err(Error::InvalidLauncher(format!(
	"Only paths with a length up to 32KBs are supported but the Python executable path has a length of {raw_length}"
	)));
	}
	// SAFETY: Above we guarantee the length is less than 32KB
	raw_length as usize
	};
	// Seek to the start of the path
	let Ok(_) = file.seek(io::SeekFrom::End(
	-((MAGIC_NUMBER_SIZE + PATH_LENGTH_SIZE + path_length) as i64),
	)) else {
	return Err(Error::InvalidLauncher(
	"Unable to seek to the start of the path".to_string(),
	));
	};
	// Read the path
	let mut buffer = vec![0u8; path_length];
	file.read_exact(&mut buffer)
	.map_err(|err| Error::InvalidLauncherRead("executable path".to_string(), err))?;
	let path = PathBuf::from(String::from_utf8(buffer).map_err(|_| {
	Error::InvalidLauncher("Python executable path was not valid UTF-8".to_string())
	})?);
	Ok(Some(Self {
	kind,
	python_path: path,
	}))
	}
	}

Notably with #8637 there isn't any zip payload at all in this variant, which perhaps simplifies things — we just won't be able to inspect the script variants.

[nahco314]

Anyway, I will read changes in #8637 .

[samypr100]

Generally speaking I'd assume you'd always want the magic at the beginning or end (not in-between) to avoid issues with the magic casually popping up in-between twice due to other reasons (e.g. particular exe ends up having the magic repeated mid-way) which can happen often depending on the magic.

[nahco314]

If we parse the EOCD and read the magic number in the middle of the file, the probability of false positives is as low as with the current method (or more).
Perhaps the problem is that the implementation would be somewhat more complicated. If we should to avoid that, padding the exe file sounds like a good idea.

[nahco314]

One idea: In the case of UVSC, embedding "UV" in the comment length section of the EOCD and using the EOCD's magic number (PK\005\006) for detection allows for a robust and reliable method that only requires static reading.
For UVPY, simply placing "UVPY" at the end is sufficient.

[nahco314]

I neglected this for a while, but I've now updated it to match the latest main!

• For now, I've kept the magic number placed before the zip content (as before). This makes the parsing process a bit more complex, but the probability of false detection shouldn’t be an issue. Essentially, it's the same in that it relies on the probability that a specific 4 bytes appears at a specific location in the file.
• As for tests, there are unit tests in uv-trampoline-builder, so I believe those are sufficient.

In the end, whether or not we need 3.7 support is up to the maintainers, but I’ve made it so that we can merge this anyway.

[T-256]

Nice, two comments that also can add them as follow-up.

[T-256]

function return type could be Result<Self, Error>.

[nahco314]

Thank you for your review and suggestion!
However, since it is LauncherKind::try_from_file that determines whether the file is actually a launcher, I believe returning Ok(None) makes sense when the file is not recognized as one.

[T-256]

I believe returning Ok(None) makes sense when the file is not recognized as one.

I think a meaningful Error variant also does the job.

-        let Ok(_) = file.seek(io::SeekFrom::End(
-            -((skip_bytes + MAGIC_NUMBER_SIZE) as i64),
-        )) else {
-            return Ok(None);
-        };
+        file.seek(io::SeekFrom::End(
+            -((skip_bytes + MAGIC_NUMBER_SIZE) as i64),
+        )).map_err(Error::InvalidLauncherSeek)?;  # <-- I think this is more proper return than `Ok(None)`

...

- Ok(Self::try_from_bytes(buffer))
+ Self::try_from_bytes(buffer).ok_or(Error::InvalidMagic)

[nahco314]

Are you suggesting that, rather than just changing LauncherKind::try_from_path, we should also implement Launcher::try_from_path so that it does not return an Option in the first place?

[nahco314]

Error handling changes a little, but the operation is fine, so I switched to this method.

[T-256]

Thanks, it looks great!
For reviewers' interests: uv somehow exposed its support of Python 3.7 via uv python where Python 3.7 is lowest version of available distributions.

[T-256]

I think it's better to have them separated to correctly show them in table:

### How do you use it as Python mirror?

Basically, this invokes the provided `python.exe` executable.

The intended use is:

- First, place our prebuilt `.exe` content at the top of the file.
- After the exe file content, write the path to the Python executable that the script uses to run
  the Python script as UTF-8 encoded string, followed by the path's length as a 32-bit little-endian
  integer.
- Write the magic number 'UVPY' in bytes.

|       `launcher.exe`        |
| :-------------------------: |
|   `<path to python.exe>`    |
| `<len(path to python.exe)>` |
| `<b'U', b'V', b'P', b'Y'>`  |

### How do you use it as Script runner?

Basically, this looks up `python.exe` (for console programs) and invokes
`python.exe path\to\the\<the .exe>`.

The intended use is:

- First, place our prebuilt `.exe` content at the top of the file.
- After the exe file content, write the path to the Python executable that the script uses to run
  the Python script as UTF-8 encoded string, followed by the path's length as a 32-bit little-endian
  integer.
- Write the magic number 'UVSC' in bytes.
- Finally, rename your Python script as `__main__.py`, compress it into a `.zip` file, and append
  this `.zip` file to the end of one of our prebuilt `.exe` files.

|       `launcher.exe`        |
| :-------------------------: |
|   `<path to python.exe>`    |
| `<len(path to python.exe)>` |
| `<b'U', b'V', b'S', b'C'>`  |
|  `<zipped python script>`   |