Getting "unexpected HTTP response code 307 Temporary Redirect" on compile.

[pineapplehunter]

Hi.
I could not download the packages when I tried to compile.

This was the output when compiling.

$ tectonic sample.tex 
note: connecting to https://archive.org/services/purl/net/pkgwpub/tectonic-default
error: unexpected HTTP response code 307 Temporary Redirect for URL https://archive.org/services/purl/net/pkgwpub/tectonic-default

I looked at the response and it seems that if you follow the link you can find the .tar file, so I think if you can follow the link this would be fixed.

[pineapplehunter]

I was looking at the code and I thing it has some thing to do with this redirect policy.
When I switched it to the default policy it seemed to work.

tectonic/crates/geturl/src/reqwest.rs

Lines 46 to 67 in cde89ce

	let redirect_policy = Policy::custom(|attempt| {
	// In the process of resolving the file url it might be necessary
	// to stop at a certain level of redirection. This might be required
	// because some hosts might redirect to a version of the url where
	// it isn't possible to select the index file by appending .index.gz.
	// (This mostly happens because CDNs redirect to the file hash.)
	if attempt.previous().len() >= MAX_HTTP_REDIRECTS_ALLOWED {
	attempt.error("too many redirections")
	} else if let Some(segments) = attempt.url().clone().path_segments() {
	let follow = segments
	.last()
	.map(|file| file.contains('.'))
	.unwrap_or(true);
	if follow {
	attempt.follow()
	} else {
	attempt.stop()
	}
	} else {
	attempt.follow()
	}
	});

[mkpankov]

Hitting this in CI as well.

[rikhuijzer]

The problem appears to be that archive.org moved some things around (307 Temporary redirect) and now Tectonic is seeing more redirects than expected

url	status
https://archive.org/services/purl/net/pkgwpub/tectonic-default	307 Temporary redirect
https://purl.prod.archive.org/net/pkgwpub/tectonic-default	302 Found
https://ttassets.z13.web.core.windows.net/tlextras-2020.0r0.tar	200 OK

[rikhuijzer]

I'm sure that this issue will be fixed sooner or later and a new release is going to be published. However, if archive.org cannot be fixed, then for new Tectonic releases to propagate to all systems can still take a while. Also, I have a 20+ minute CI job which is flaky due to archive.org being often non-responsive (#765).

Therefore, I've setup a new redirect at https://tectonicredirectmirrorabc.netlify.app/tectonic-default. This URL is the same length as the URL embedded in the binary. Therefore, a simple sed -i 's@old@new@' tectonic can patch it:

#!/usr/bin/env bash

OLDURL="https://archive.org/services/purl/net/pkgwpub/tectonic-default"
NEWURL="https://tectonicredirectmirrorabc.netlify.app/tectonic-default"

# Patch the Tectonic binary.
sed -i "s@$OLDURL@$NEWURL@" tectonic

Thanks to Netlify for allowing me to easily pick a subdomain name and for setting HTTP 302 redirects (I've tried refreshes via HTML, but those were ignored by the HTTP library). The Netlify configuration is at https://github.com/JuliaBooks/TectonicRedirect/.

[pkgw]

@rikhuijzer Thank you very much for your initiative in working around this! Based on how Tectonic's redirection code is supposed to work, this change in archive.org shouldn't be causing problems, I think, but apparently it is.

With there also being the issue that archive.org doesn't work in China, this may be the thing that pushes me over the edge to register a domain and set up our own infrastructure for managing this step of the process. Depending on how long that takes, it may still be worthwhile to put out a release that works around the archive.org issue, but if old binaries are going to be broken, I'm inclined to go for the full solution.

[rikhuijzer]

Based on how Tectonic's redirection code is supposed to work, this change in archive.org shouldn't be causing problems, I think, but apparently it is.

Yes, I'm now very sure that it is. In my package, I now patch and try again if the PDF build fails. From the build logs in my package:

Wrote _build/books.tex (for debugging purposes)
error: unexpected HTTP response code 307 Temporary Redirect for URL https://archive.org/services/purl/net/pkgwpub/tectonic-default
Error producing PDF.
note: reading from standard input; outputs will appear under the base name "texput"
note: connecting to https://archive.org/services/purl/net/pkgwpub/tectonic-default

PDF generation failed. Patching the archive.org URL and trying again
Wrote _build/books.tex (for debugging purposes)
[...]
Built _build/books.pdf

With there also being the issue that archive.org doesn't work in China, this may be the thing that pushes me over the edge to register a domain and set up our own infrastructure for managing this step of the process. Depending on how long that takes, it may still be worthwhile to put out a release that works around the archive.org issue

Feel free to copy my Netlify configuration. It's very easy to attach Netlify to a domain and also hosting is for free. I expect that Netlify is also a lot more stable than archive.org, so it would solve #765.

Maybe, even better would be to add two links in the binary. If the first one fails, then try the second one.

[pkgw]

I think the existing problem is an issue in reqwest, filed as seanmonstar/reqwest#1348.

I am working on setting up a dedicated redirection service, which should hopefully be live soon — the limiting factor has just been me getting the DNS straightened out.

I think it is actually better not to have two links in the binary. The service needs to be reliable, and adding multiple URL options just diffuses the responsibility for making sure that the service is working.

[pkgw]

Actually, no. The problem is in our custom redirection logic. There's at least one issue: Tectonic follows redirects as long as the new URL path has a final component that contains a ., with the idea that this recognizes redirects that look like filenames, but stops when we get to S3-type hashed storage. The new redirect stage from archive.org preserves the structure of our PURL URL, /net/pkgwpub/tectonic-default, which happens to not include a dot, so we immediately stop chasing redirects. This is definitely wrong behavior in this case.

It's also true that once we stop chasing redirects, we require that the most recent HTTP response code is either a 200 series or a 302, but something like a 307 is considered a failure. Based on how 307 is supposed to be essentially a more precisely-specified version of 302, we should almost surely accept it as a final result, although in this particular case it would have masked incorrect behavior.

I think I am going to change the redirect-chasing logic to also follow a redirect if the final path element of the new URL is the same as the final path element of the original URL. This seems like a good heuristic for ignoring S3-type redirects that ought to work for this particular case.

[pkgw]

Addressed in #832. I still hope to soon issue a new release that will include not only this fix, but an update of the builtin URL to a new service that will fix some of the other issues that have been associated with the archive.org PURL service.

[pkgw]

I've merged in #832, so binaries built from master (or, shortly, the latest continuous-deployment release) should be functional again.

[pkgw]

I have posted #833 to updated the builtin bundle URL to a custom domain and service that I set up. Testing welcomed! The proposed new bundle URL is https://relay.fullyjustified.net/default_bundle.tar.

[pkgw]

#833 is now merged, so master and the latest continuous-deployment release will start using the new builtin default URL.

[pkgw]

A new Tectonic release, 0.8.0, is now processing in the CI systems, and should be finalized in a bit more than an hour.