Failed: Network error: error sending request for url

[raffaem]

❯ lychee --verbose --user-agent 'curl/7.54' _posts/2023-04-22-WOEPSR23.md
✗ [ERR] https://www.ip.mpg.de/en/research/innovation-and-entrepreneurship-research/woepsr2023.html | Failed: Network error: error sending request for url (https://www.ip.mpg.de/en/research/innovation-and-entrepreneurship-research/woepsr2023.html)
✔ [200] https://www.kth.se/profile/andbr

Issues found in 1 input. Find details below.

[_posts/2023-04-22-WOEPSR23.md]:
✗ [ERR] https://www.ip.mpg.de/en/research/innovation-and-entrepreneurship-research/woepsr2023.html | Failed: Network error: error sending request for url (https://www.ip.mpg.de/en/research/innovation-and-entrepreneurship-research/woepsr2023.html)

🔍 2 Total (in 1s) ✅ 1 OK 🚫 1 Error
[ble: exit 2]

Contents of _posts/2023-04-22-WOEPSR23.md:

---
layout: post
title: WOEPSR23 Presentation
date: 2023-04-22 08:46:00+0200
description: paper presentation at WOESPR23
tags: ["conference", "publications"]
categories: conference
related_posts: false
---

On the 13th of April 2023, I presented my paper "Do Mission-Oriented Grant Schemes Shape the Direction of Science?", co-authored with [Anders Broström](https://www.kth.se/profile/andbr), at the [WOEPSR23](https://www.ip.mpg.de/en/research/innovation-and-entrepreneurship-research/woepsr2023.html) conference.

Thanks again for the valuable suggestions!

{% include figure.liquid loading="eager" path="assets/img/2023-04-22-WOEPSR23.JPG" class="img-fluid rounded z-depth-1" zoomable=true %}

Related: #733

[mre]

Hm, works for me:

echo 'https://www.ip.mpg.de/en/research/innovation-and-entrepreneurship-research/woepsr2023.html' | lyc
hee -
  1/1 ━━━━━━━━━━━━━━━━━━━━ Finished extracting links
🔍 1 Total (in 3s) ✅ 1 OK 🚫 0 Errors

Do you run this locally?

In the issue you linked, we had a similar problem: it worked on my end, but didn't work for others.
We never figured out why. (I'm on a mac.)

Does this work for you?

curl -vvv https://www.ip.mpg.de/en/research/innovation-and-entrepreneurship-research/woepsr2023.html

[raffaem]

Do you run this locally?

Yes

Does this work for you?

No

➜  curl -vvv https://www.ip.mpg.de/en/research/innovation-and-entrepreneurship-research/woepsr2023.html
* Host www.ip.mpg.de:443 was resolved.
* IPv6: (none)
* IPv4: 193.174.132.100
*   Trying 193.174.132.100:443...
* Connected to www.ip.mpg.de (193.174.132.100) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* closing connection #0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the webpage mentioned above.
[ble: exit 60]

[mre]

Looks like a certificate error. Try this: https://stackoverflow.com/a/24618403/270334
Which platform are you on?

[mre]

If you're on Debian/Ubuntu, this might help:

sudo apt-get install ca-certificates

[raffaem]

No, looks like the server doesn't send over the intermediate certificate.

Desktop browsers are able to fill the missing certificate, but tools like curl and probably lychee expect the full chain

See::

https://security.stackexchange.com/questions/172989/checking-intermediate-certificates-programatically
https://security.stackexchange.com/questions/187313/chain-of-trust-incomplete-and-server-cipher-order-not-set

[thomas-zahner]

On my machine (Arch Linux) I can reproduce the problem.

The following is mentioned on the URL provided in curl's error message. This would explain why @mre can't reproduce the problem.

If libcurl was built with Schannel or Secure Transport support (the native SSL libraries included in Windows and macOS), then this does not apply to you.

So it seems like the Windows and macOS libraries trust the certificates whereas libcurl (and probably reqwest) don't. On my machine I can get both lychee and curl to work with the insecure flag:

curl -vvv https://www.ip.mpg.de/en/research/innovation-and-entrepreneurship-research/woepsr2023.html --insecure

echo 'https://www.ip.mpg.de/en/research/innovation-and-entrepreneurship-research/woepsr2023.html' | lychee - --insecure

So to me this does not seem like a bug in lychee.

[mre]

Very interesting! Thanks for digging that up.
Yeah, I agree that it's more of a design decision on different platforms.

I documented some tips and tricks at https://lychee.cli.rs/troubleshooting/network-errors/

Perhaps we could also make the error message a bit more self-explanatory.
E.g.

Failed: Network error: error sending request for url... Maybe this is a certificate error?

[mre]

This will be fixed with #1521.