Weird SSL(?) error

[Schnobs]

CKAN Version: 1.24.0
KSP Version: 1.3.1
Operating System: Debian 9
Have you made any manual changes to your GameData folder (i.e., not via CKAN)? nope

Problem

Tried to install Realism Overhaul for KSP1.3.1.

That's a lot of mods, some of them come in rather large files. CKAN downloaded most files over the course of about 10 minutes, then stalled. I cancelled the install, and started over from the CKAN main menu, doing the same selections as before. Most mods were cached, but the three still outstanding downloads were failed with an certificate error.

Re-importing certificates as suggested in https://github.com/KSP-CKAN/CKAN/wiki/SSL-certificate-errors did not help (ca-certificates-mono is already the newest version, cert-sync has nothing to do).

It strikes me as odd that everything works fine for a while, then throws SSL errors two minutes later. The certificate on the github website is valid until 17-May-2018 (only six weeks to go); I don't think it has been replaced just now. I'm suspecting some load-balancing or abuse prevention mechanism is at work, hence the question mark in the title.

FWIW, here's the full output of the second (failed) download attempt. I'd like to emphasisze once more that all all "cached" files were downloaded just a few minutes before:

About to install...

 * Realism Overhaul v12.0.0 (cached)
 * Deadly Reentry Continued v7.6.2 (cached)
 * HangerExtender 3.5.2 (cached)
 * KSP AVC 1.1.6.2 (cached)
 * MechJeb 2 2.7.0.0 (cached)
 * Procedural Fairings v4.0 (cached)
 * Procedural Parts v1.2.14 (cached)
 * PersistentRotation 1.8.4 (cached)
 * KSC Switcher 0.8 (cached)
 * Realism Overhaul Craft Files v12.0.0 (cached)
 * RemoteTech v1.8.9 (cached)
 * TAC Life Support (TACLS) v0.13.6.0 (cached)
 * TextureReplacerReplaced V0.5.4 (cached)
 * Toolbar 1.7.16.5 (cached)
 * Ven's Stock Part Revamp v1.9.6 (cached)
 * DMagic Orbital Science 1.3.11 (cached)
 * Kerbal Attachment System 0.6.3.0 (cached)
 * SCANsat v18.4 (cached)
 * FASA 1:v7.2.2 (cached)
 * SXTContinued 1:0.3.23.1 (cached)
 * Advanced Jet Engine v2.10.0 (cached)
 * Ferram Aerospace Research 3:0.15.9.1 (github.com, 751.9 KB)
 * Solver Engines plugin v3.5 (cached)
 * Kerbal Joint Reinforcement v3.3.3 (cached)
 * RealChute Parachute Systems v1.4.5 (github.com, 2.6 MB)
 * Real Fuels rf-v12.5.0 (cached)
 * Community Resource Pack 0.8.1.0 (cached)
 * RealHeat v4.5 (cached)
 * Real Plume 2:v11.0.0 (cached)
 * SmokeScreen - Extended FX Plugin 2.7.6.1 (cached)
 * Background Resources v0.13.6.0 (cached)
 * Firespitter Core v7.6.0 (github.com, 29.2 MB)
 * Firespitter Resources config v7.6.0 (github.com, 29.2 MB)
 * Retractable Lifting Surface Module 0.1.5 (cached)

Downloading "https://github.com/ferram4/Ferram-Aerospace-Research/releases/download/v0.15.9.1_Liepmann/FAR_0_15_9_1_Liepmann.zip"
Downloading "https://github.com/StupidChris/RealChute/releases/download/v1.4.5/RealChute_v1.4.5.zip"
Downloading "https://github.com/snjo/Firespitter/releases/download/v7.6.0/Firespitter_7.6.0.zip"
Oh no! Our download failed with a certificate error!

Consult this page for help:
	https://github.com/KSP-CKAN/CKAN/wiki/SSL-certificate-errors
Error!

[HebaruSan]

Hmm, I wonder whether GitHub throttling can manifest as a cert error. Usually it's a 403 forbidden response, but if you just downloaded a ton of files from GitHub, throttling would become more and more likely. May be a duplicate of #1817 or #2210 if that's the cause. We have some changes in progress to address the throttling, but they're not complete yet.

Also, I note that these three modules specifically are listed as "license": "restricted". If they were free licenses, they would have fallback URLs on archive.org (see #2284), which may be how the rest of the list was able to succeed after GitHub gets fed up.

Sorry, that's a long-winded way of saying to try waiting at least an hour or two before trying again.

[Schnobs]

After messing around with CA and Intermediate certificates for a while, I left it alone and had a nice day. Now I'm back after several hours and started over from a almost clean slate: fresh KSP install, introduce it to CKAN, and so on.

Only variation was that I kept all files >5MB and dumped them into the download directory ahead of time.

CKAN fetched most of the the rest, but balked at the very same files as before.

[Schnobs]

Incidentally, how does CKAN arrive at the eight-digit number it prepends to the downloaded files' names? And, seeing as the registry.json file already contains checksums, what's the purpose of these filename additions?

[HebaruSan]

That's the first eight characters in the hexadecimal interpretation of the SHA1 of the download URL. Its purpose is to map URLs to cached files, so the client can decide whether to download a URL or not.

CKAN/Core/Net/NetFileCache.cs

Lines 348 to 357 in a495f43

	// returns the 8-byte hash for a given url
	public static string CreateURLHash(Uri url)
	{
	using (var sha1 = new SHA1Cng())
	{
	byte[] hash = sha1.ComputeHash(Encoding.UTF8.GetBytes(url.ToString()));
	return BitConverter.ToString(hash).Replace("-", "").Substring(0, 8);
	}
	}

[Schnobs]

If they were free licenses, they would have fallback URLs on archive.org (see #2284), which may be how the rest of the list was able to succeed after GitHub gets fed up.

Does ckan do (or has a switch for) extended logging to find out, or would I have to observe a session using netstat or somesuch?

Also, from browsing the issues i gather that SSL issues won't get away until mono5 becomes commonplace. Seeing as I for one don't seem to have problems when using curl or wget, I wonder if CKAN could use either as fallback mechanism?

[HebaruSan]

Does ckan do (or has a switch for) extended logging to find out, or would I have to observe a session using netstat or somesuch?

Sure, if you're running from the command line you can use the --verbose flag to enable this message:

CKAN/Core/Net/NetAsyncDownloader.cs

Line 446 in a495f43

log.InfoFormat("Trying fallback URL: {0}", downloads[index].fallbackUrl);

Or if you're in GUI, you can enable it with a log config file:
https://github.com/KSP-CKAN/CKAN/wiki/User-guide#logging

Also, from browsing the issues i gather that SSL issues won't get away until mono5 becomes commonplace. Seeing as I for one don't seem to have problems when using curl or wget, I wonder if CKAN could use either as fallback mechanism?

Currently no, there's built-in no way to use an external program as a downloader. Remember that CKAN is cross platform and most Windows installs don't have wget or curl natively available. However, if you download manually with wget or curl, it is possible to import those downloaded files into CKAN, see #2246 and #2264.

[Schnobs]

it is possible to import those downloaded files

Palm, meet face. I didn't know that and manually prepended my downloads with the right hash (hence the question above)... thanks.

[Schnobs]

Alright, I had no luck with logging but tcpdump leaves no doubt: I'm downloading my packages from archive.org. If it's not there, I don't get it.

[HebaruSan]

Which implies that your Mono either doesn't support TLS1.2 or doesn't have the certificates it needs (similar to #2278). I was afraid that might be it, but I wanted to rule out the other possibilities. Unfortunately I don't know anything more about certs than is on the wiki.

You're welcome for the fallback downloads, though; without #2284, you wouldn't get any downloads at all!

[HebaruSan]

We've had two more recent investigations that revealed specific things that can go wrong with Mono. They may or may not relate to the issue you've been having:

• Update your version of libmono-*security
• Update the files in /etc/mono with your OS's packaging system