Flexo panics with custom repo

[pidario]

Before I start telling you about the issue I have to say that my setup is probably quite uncommon:

I want to run flexo on a raspberry pi 2 (at the moment it runs raspbian buster, eventually I want to install Arch ARM).
For some reason, the pi could not compile flexo: cargo seg faults without other message. So I decided to cross compile it for armv7 from my main PC (running standard arch linux) using cross.

cross build --release --target armv7-unknown-linux-gnueabihf

Building is fine (but at first it exit with an error because of some openssl header missing so as suggested here I added the following to Cargo.toml

openssl = { version = "0.10", features = ["vendored"] }

and everything went fine. I then moved the built binary to my Pi, created the configuration file at /etc/flexo/flexo.toml using this. And finally on my Arch PC I added Server = http://192.168.1.30:7878/$repo/os/$arch to my /etc/pacman.d/mirrorlist file.

Everything works fine, until I decide to add a custom repo.
Without flexo /etc/pacman.conf has this custom repo, and works just fine:

[home_ungoogled_chromium_Arch]
SigLevel = Required TrustAll
Server = https://download.opensuse.org/repositories/home:/ungoogled_chromium/Arch/$arch

after flexo, that repo becomes:

[home_ungoogled_chromium_Arch]
SigLevel = Required TrustAll
Server = http://192.168.1.30:7878/custom_repo/$repo/$arch

and I appended this to the flexo configuration file:

[[custom_repo]]
name = "home_ungoogled_chromium_Arch"
url = "https://download.opensuse.org/repositories/home:/ungoogled_chromium/Arch

Then when I try to install ungoogle-chromium via pacman -S ungoogled-chromium pacman exit because it cannot find the file (err 404) and flexo crashes with this stack trace:

thread '<unnamed>' panicked at 'called `Option::unwrap()` on a `None` value', src/mirror_flexo.rs:394:61
stack backtrace:
   0:   0x6d7e44 - std::backtrace_rs::backtrace::libunwind::trace::h41badf34901cec2e
                       at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/../../backtrace/src/backtrace/libunwind.rs:90:5
   1:   0x6d7e44 - std::backtrace_rs::backtrace::trace_unsynchronized::hda5d46c6d89af569
                       at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:   0x6d7e44 - std::sys_common::backtrace::_print_fmt::h116b0f7c44be2f72
                       at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/sys_common/backtrace.rs:67:5
   3:   0x6d7e44 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hae65de3082032db4
                       at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/sys_common/backtrace.rs:46:22
   4:   0x6fe7b4 - core::fmt::write::h3bc01e3bc08b8381
                       at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/core/src/fmt/mod.rs:1115:17
   5:   0x6d2bd4 - std::io::Write::write_fmt::he73de01764266ae1
                       at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/io/mod.rs:1665:15
   6:   0x6da620 - std::sys_common::backtrace::_print::h5736d994d09b7d24
                       at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/sys_common/backtrace.rs:49:5
   7:   0x6da620 - std::sys_common::backtrace::print::h2b203eb5a03fa719
                       at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/sys_common/backtrace.rs:36:9
   8:   0x6da620 - std::panicking::default_hook::{{closure}}::hc3121852063d883d
                       at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:208:50
   9:   0x6da008 - std::panicking::default_hook::h37c886f9c8759435
                       at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:225:9
  10:   0x4d76e0 - flexo::main::{{closure}}::h9ce357dfabb626b6
  11:   0x6dad38 - std::panicking::rust_panic_with_hook::ha5b17f06e1496c69
                       at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:626:17
  12:   0x6da7ac - std::panicking::begin_panic_handler::{{closure}}::h5084db84532a4dae
                       at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:517:13
  13:   0x6d83fc - std::sys_common::backtrace::__rust_end_short_backtrace::h08c7764c0a0c04ef
                       at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/sys_common/backtrace.rs:141:18
  14:   0x6da72c - rust_begin_unwind
                       at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:515:5
  15:   0x47ffb0 - core::panicking::panic_fmt::hd56493e17d6e29d1
                       at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/core/src/panicking.rs:92:14
  16:   0x47ff0c - core::panicking::panic::he7ce3ec96e9efb28
                       at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/core/src/panicking.rs:50:5
  17:   0x4a82b8 - flexo::Order::try_until_success::he9cd54dc6af74fd4
  18:   0x4e80b4 - std::sys_common::backtrace::__rust_begin_short_backtrace::h966176c1005035c2
  19:   0x4ea7a8 - core::ops::function::FnOnce::call_once{{vtable.shim}}::hae5e5b551d65bcdb
  20:   0x6e0454 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h5d8d110a8c21a463
                       at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/alloc/src/boxed.rs:1572:9
  21:   0x6e0454 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h0ddaf8a69a7ac79d
                       at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/alloc/src/boxed.rs:1572:9
  22:   0x6e0454 - std::sys::unix::thread::Thread::new::thread_start::h2935281fd82b26fa
                       at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/sys/unix/thread.rs:74:17

Installing other packages from core extra or community works fine even after adding the custom repo.

The strange thing is that if I try to manually download other files from the custom_repo through the proxy, it works:
wget http://192.168.1.30:7878/custom_repo/home_ungoogled_chromium_Arch/x86_64/ungoogled-chromium-95.0.4638.69-1-x86_64.pkg.tar.zst.sig
But trying to download the package archive with
wget http://192.168.1.30:7878/custom_repo/home_ungoogled_chromium_Arch/x86_64/ungoogled-chromium-95.0.4638.69-1-x86_64.pkg.tar.zst
causes flexo to crash with the same stack trace.
Clearly the file exists in the repo

Do you know what's the deal? Some shenanigans with the binary built that way? Or maybe because of the OS? or the architecture?

Thanks in advance for taking your time to give a look to this issue and for developing such an amazing piece of software

[pidario]

Update:

I forgot to mention that I'm using version 1.6.2.

After launching flexo with RUST_LOG=debug env variable (maybe this information should be in the README), I clearly see that the program outputs a lot of lines like the following:

[2021-11-09T13:59:58.163Z DEBUG flexo::mirror_flexo] File unavailable - return content length without writing anything.

and as the last line before crashing:

[2021-11-09T13:59:58.163Z DEBUG flexo::mirror_flexo] https://download.opensuse.org/repositories/home:/ungoogled_chromium/Arch replied with status code 200.

Launching with the info log level gives this before crashing:

[2021-11-09T14:01:40.364Z INFO  flexo] Request served [NO PAYLOAD]: "custom_repo/home_ungoogled_chromium_Arch/x86_64/ungoogled-chromium-95.0.4638.69-1-x86_64.pkg.tar.zst"

[nroi]

Thank you for your bug report.

I was able to reproduce the issue on x86, so the issue is not related to ARM. But describing your ARM setup was helpful, I wasn't aware that such changes are required to build Flexo using cross.

The issue seems to be related to redirects. The url https://download.opensuse.org/repositories/home:/ungoogled_chromium/Arch/x86_64/ungoogled-chromium-95.0.4638.69-1-x86_64.pkg.tar.zst redirects me to this url, which, in turn, redirects me to another url, which, finally, redirects me to this url. If I use the final destination in my flexo.toml, it works. So, in my case, I add:

[[custom_repo]]
name = "home_ungoogled_chromium_Arch"
url = "https://widehat.opensuse.org/repositories/home:/ungoogled_chromium/Arch"

and then I can successfully download the file via Flexo:

curl -vv http://127.0.0.1:7878/custom_repo/home_ungoogled_chromium_Arch/x86_64/ungoogled-chromium-95.0.4638.69-1-x86_64.pkg.tar.zst > /dev/null

You may try this as a workaround, but I wouldn't just copy-paste the url from my config, but instead see where you're redirected to (perhaps the mirror is using geolocation-based redirects or something fancy). You can use curl to determine the final location:

curl -Ls -o /dev/null -w %{url_effective} https://download.opensuse.org/repositories/home:/ungoogled_chromium/Arch/x86_64/ungoogled-chromium-95.0.4638.69-1-x86_64.pkg.tar.zst

But that's obviously just a workaround until Flexo can handle redirects properly.

[pidario]

Glad to be helpful about the ARM build.

Anyway, that did the trick, thanks. The final URL was the same for me.
Is a fix in your immediate plan? If not, I might be able to find the time to make a PR.

[nroi]

Is a fix in your immediate plan? If not, I might be able to find the time to make a PR.

I'm planning to look into it this weekend, I can't say how long it's going to take.