[fxa-email-service] fail to install

[karansapolia]

Everytime I try 'npm install' in fxa, it fails on:

+ cd fxa-email-service
+ cargo build --bin fxa_email_send
   Compiling openssl v0.9.24
error: failed to run custom build command for `openssl v0.9.24`
process didn't exit successfully: `/home/x/fxa/packages/fxa-email-service/target/debug/build/openssl-0e5b276e20c12757/build-script-build` (exit code: 101)
--- stderr
thread 'main' panicked at 'Unable to detect OpenSSL version', /home/x/.cargo/registry/src/github.com-1ecc6299db9ec823/openssl-0.9.24/build.rs:16:14
note: Run with `RUST_BACKTRACE=1` for a backtrace.

warning: build failed, waiting for other jobs to finish...
error: build failed

I have openssl-devel and pkg-config installed. Any workarounds?

[shane-tomlinson]

@karansapolia - are you using Rust nightly? It's easy to miss in the list of dependencies and use the Release version of Rust.

[karansapolia]

@karansapolia - are you using Rust nightly? It's easy to miss in the list of dependencies and use the Release version of Rust.

$ rustc --version
rustc 1.37.0-nightly (02564de47 2019-06-10)

Did a 'rustup update' yesterday to see if it works with the latest nightly

[karansapolia]

Any word on this? This is still an issue. I work using 'npm run start-remote' in the fxa-content-server folder because that's where I mostly contribute to. But 'npm i' on fxa always fails, even though I use rust nightly.

[wavepack3t]

After quite some time trying to get this to work I think the root of the problem is dependency breakdowns between different older versions of rust libraries and OpenSSL.

Anyone with more recent versions of OpenSSL is at risk of running into issues. For example I am running

$ openssl version
OpenSSL 1.1.1 11 Sep 2018

There was an issue a while back with reqwest which used an old version of the Rust openssl library, openssl = 0.9.24.
seanmonstar/reqwest#345

This problem was fixed when reqwest updated their Cargo.toml to use a newer version, openssl 0.10.X (by updating their native-tls dependency). This change was made in this release https://github.com/seanmonstar/reqwest/releases/tag/v0.9.0 so libraries using reqwest 0.9 or above should be fine.

However, fxa-email-service still depends on reqwest 0.8.5 as you can see here in master: https://github.com/mozilla/fxa/blob/master/packages/fxa-email-service/Cargo.toml#L38

I tried forking this project and upgrading reqwest but there are still many dependencies which cause openssl 0.9 to be installed. It's a real mess... I gave up on fixing this a few days ago.

[karansapolia]

@wavepack3t I too have openssl version 1.1.1 installed. So as a workaround, should I install an older version of openssl on my system? version 0.9.24?

[wavepack3t]

I guess you could try downgrading to 1.0.1 as that is what is supported by the Rust openssl 0.9.24 release...

rust-openssl depends on OpenSSL version 1.0.1 or above, or LibreSSL
https://github.com/sfackler/rust-openssl/tree/openssl-v0.9.24

Downgrading def feels like the greater of two evils from a personal security standpoint... Let me know if it works though! :)

Of course the ideal fix is to update the fxa-email-service dependencies to no longer depend on openssl 0.9.24.

[philbooth]

Somewhat related to #921 (comment), where I started work on a branch that updates our ancient version of Rust and all of the dependencies, then tried to work through all of the knock-on issues that fell out from there.

That effort got shelved because it was taking me a while to work through all the issues from updating everything, but presumably it's good to pick up again if this is actually blocking people from working locally now.

[philbooth]

/cc @clouserw, I've assigned this to myself and moved it to "in progress" but let me know if that's wrong.

[clouserw]

Thanks @philbooth . Please fill in the complexity estimate so we can figure out timing in our schedule.

[bbangert]

In one of my other PR's I disabled the build of email-service due to this. I think it will require an update before it can be built again in a docker container as well, since the underlying issue is that it cannot build against the latest openssl until those Rust crate dependencies are updated.

[philbooth]

Just noting that the build problems are all resolved in the pb/921-cargo-audit branch, I'm just working through a few unit test breakages that are a result of the updates.

Ref: rwf2/Rocket#1040

[wavepack3t]

@philbooth
I checked out pb/921-cargo-audit just now and upon build attempt I got the same openssl error. What version of OpenSSL is in the build environment?

I notice that you have updated rocket in your branch (thank you!) but your version of reqwest remains the same, so the old rust openssl 0.9 dep still gets pulled.

Here's a broken branch of mine where I have updated reqwest but have not updated rocket... maybe it will help as a reference in some way? Needed to do some style changes in fxa-email-service to at least get the reqwest upgrade to work. https://github.com/wavepack3t/fxa/tree/update-deps Warning: does not merge cleanly with your branch

[philbooth]

but your version of reqwest remains the same

Nope, I did update it, see the diff:

+ "reqwest 0.9.18 (registry+https://github.com/rust-lang/crates.io-index)",

However, looks like 0.8 is still in the lockfile as a dependency of socketlabs and mozsvc-common though. I'll see about getting those updated too.

[wavepack3t]

My mistake, I am new to Rust and got confused since the Cargo.toml entry for reqwest didn't change... I did not think to check the Cargo.lock under fxa-email-service.

As far as updating socketlabs / mozsvc-common... that's where I gave up.

socketlabs hasn't been updated in 2018... you could try to reach out to the author but I doubt you'll get a reply in a timely manner. Maybe you'll have better luck with mozsvc-common as https://github.com/mozilla-services/common-rs was updated May of this year.

I forked both packages and forced them to use reqwest 0.9.4 thinking that would do the trick.

https://github.com/wavepack3t/socketlabs-rs
https://github.com/wavepack3t/common-rs

Unfortunately I had to publish those under different crate names because I am not the crate owner. Don't really know how this sort of thing is handled in the Rust community. In any case, I do get a successful build when I use my two janky renamed crates and rename their corresponding import statements in the code!

$ cargo build
   Compiling fxa_email_service v1.140.1 (...fxa/packages/fxa-email-service)
    Finished dev [unoptimized + debuginfo] target(s) in 31.36s

Advice: do not use these crates :)
https://crates.io/crates/mozsvc-common-wvp
https://crates.io/crates/socketlabz

Anyway just posting for your info. Thank you for devoting attention to this issue! I would very much like to get my own fxa server running off of origin master.

[vladikoff]

socketlabs hasn't been updated in 2018... you could try to reach out to the author but I doubt you'll get a reply in a timely manner.

We maintain socketlabs with @brizental , new version will be published in 1-2 days

[karansapolia]

This is still an issue.

   Compiling want v0.0.4
   Compiling want v0.0.6
   Compiling lock_api v0.1.5
   Compiling tokio-executor v0.1.7
   Compiling crossbeam-queue v0.1.2
   Compiling crossbeam-channel v0.3.8
   Compiling rustc_version v0.2.3
   Compiling serde_test v0.8.23
   Compiling encoding v0.2.33
error: failed to run custom build command for `openssl-sys v0.9.47`

Caused by:
  process didn't exit successfully: `/home/x/fxa/packages/fxa-email-service/target/debug/build/openssl-sys-56117d6f23a56417/build-script-main` (exit code: 101)
--- stdout
cargo:rustc-cfg=const_fn
cargo:rerun-if-env-changed=X86_64_UNKNOWN_LINUX_GNU_OPENSSL_LIB_DIR
X86_64_UNKNOWN_LINUX_GNU_OPENSSL_LIB_DIR unset
cargo:rerun-if-env-changed=OPENSSL_LIB_DIR
OPENSSL_LIB_DIR unset
cargo:rerun-if-env-changed=X86_64_UNKNOWN_LINUX_GNU_OPENSSL_INCLUDE_DIR
X86_64_UNKNOWN_LINUX_GNU_OPENSSL_INCLUDE_DIR unset
cargo:rerun-if-env-changed=OPENSSL_INCLUDE_DIR
OPENSSL_INCLUDE_DIR unset

Complete output at: https://pastebin.com/5VZaJxqT

[philbooth]

This is interesting:

running: "cc" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-E" "build/expando.c"
cargo:warning=build/expando.c:4:24: error: pasting "RUST_VERSION_OPENSSL_" and "(" does not give a valid preprocessing token
cargo:warning=    4 | #define VERSION2(n, v) RUST_VERSION_##n##_##v
cargo:warning=      |                        ^~~~~~~~~~~~~
cargo:warning=build/expando.c:5:23: note: in expansion of macro ‘VERSION2’
cargo:warning=    5 | #define VERSION(n, v) VERSION2(n, v)
cargo:warning=      |                       ^~~~~~~~
cargo:warning=build/expando.c:10:1: note: in expansion of macro ‘VERSION’
cargo:warning=   10 | VERSION(OPENSSL, OPENSSL_VERSION_NUMBER)
cargo:warning=      | ^~~~~~~
exit code: 1

IIUC, it looks like the value of OPENSSL_VERSION_NUMBER there is (, which is pretty weird. Not sure how that's set yet, going to look into it, but in the meantime @karansapolia, what does it say when you run openssl version?

[philbooth]

@karansapolia, looks like you might be running into sfackler/rust-openssl#1030.

Did you build openssl from source or install it from package?

[karansapolia]

@karansapolia, what does it say when you run openssl version?

[x@localhost fxa]$ openssl version
openssl: error while loading shared libraries: libssl.so.3: cannot open shared object file: No such file or directory

I uninstalled the openssl-devel package and installed version 1.0.1 from source along @wavepack3t 's suggestion:

I guess you could try downgrading to 1.0.1 as that is what is supported by the Rust openssl 0.9.24 release...

I must've messed up somethings while installing. Maybe that's why I am seeing the installation error.

[philbooth]

Yeah, worth trying to do a clean installation from your package manager, hopefully that will get you back up and running.

[karansapolia]

Yeah, worth trying to do a clean installation from your package manager, hopefully that will get you back up and running.

[x@localhost fxa]$ openssl version
OpenSSL 1.1.1c FIPS  28 May 2019

Hey @philbooth, so I did a clean install of the openssl-devel package and tried 'npm install' in the 1569-old-openssl branch locally. Still fails with the same errors. I think cargo is still receiving the openssl version as "(" :

running: "cc" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-E" "build/expando.c"
cargo:warning=build/expando.c:4:24: error: pasting "RUST_VERSION_OPENSSL_" and "(" does not give a valid preprocessing token
cargo:warning=    4 | #define VERSION2(n, v) RUST_VERSION_##n##_##v
cargo:warning=      |

Complete log: https://pastebin.com/sTaB688u

[philbooth]

@karansapolia, I wonder if your system is in the same state as the one from this thread:

sfackler/rust-openssl#1037 (comment)

i.e., the headers in /usr/local/include/openssl (or wherever) are still from the version you built, so when we compile that's what is picked up rather than what's reported by the openssl binary. However, I'm not enough of a C/Linux/OpenSSL expert to know what the fix is, other than suggesting you try whatever means there are to clear them out safely.

(and I'm explicitly not telling you to delete anything in particular, because people telling other people to delete things from their computer is dangerous 😕)

[philbooth]

Fwiw, @bbangert mentioned in Slack that #1654 seemed to fix this for him, which is nice (but we can still keep this open while we get to the bottom of @karansapolia's issue).

[karansapolia]

@philbooth seems like the issue is solved. I resolved the headers issue. The fxa-email-service build is completing now on master itself. The complete install is failing due to other reasons now, I believe.
Log: https://pastebin.com/63hNpxQq

[philbooth]

Great, thanks @karansapolia, I'll close this.

The complete install is failing due to other reasons now, I believe.
Log: https://pastebin.com/63hNpxQq

Looks like Docker isn't running?