Core doesn't build on RPI 3B

[EchedelleLR]

Tested on master at this post time.

Compiling try_from v0.3.2
Compiling backtrace-sys v0.1.32
Compiling openssl-src v111.6.1+1.1.1d
Compiling num-integer v0.1.41
error[E0463]: can't find crate for cfg_if
--> /home/echedey/.cargo/registry/src/github.com-1ecc6299db9ec823/try_from-0.3.2/src/lib.rs:4:1
|
4 | extern crate cfg_if;
| ^^^^^^^^^^^^^^^^^^^^ can't find crate

error: aborting due to previous error

For more information about this error, try rustc --explain E0463.
error: could not compile try_from.
warning: build failed, waiting for other jobs to finish...
error: build failed
Traceback (most recent call last):
File "install_python_bindings.py", line 24, in
"cargo", "build", "-p", "deltachat_ffi", "--" + target
File "/usr/lib/python3.7/subprocess.py", line 347, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cargo', 'build', '-p', 'deltachat_ffi', '--release']' returned non-zero exit status 101.
(bot) echedey@debian-pi-3b:~/deltachat-core-rust/python$ rustc --explain E0463
A plugin/crate was declared but cannot be found. Erroneous code example:

#![feature(plugin)]
#![plugin(cookie_monster)] // error: can't find crate for `cookie_monster`
extern crate cake_is_a_lie; // error: can't find crate for `cake_is_a_lie`

You need to link your code to the relevant crate in order to be able to use it
(through Cargo or the -L option of rustc example). Plugins are crates as
well, and you link to them the same way.

[link2xt]

The cfg-if crate is even in our Cargo.lock so it is very strange that it couldn't be found.

Does cargo check or cargo build work in the root (not building FFI)?

[EchedelleLR]

As showed in that is written as "cfg_if". Is there any difference with it?.

[EchedelleLR]

I will have to wait:

echedey@debian-pi-3b:~/deltachat-core-rust$ cargo check
Compiling proc-macro2 v1.0.6
Compiling unicode-xid v0.2.0
Compiling syn v1.0.11
Compiling libc v0.2.66
Checking cfg-if v0.1.10
Compiling autocfg v0.1.7
Compiling serde v1.0.104
Compiling cc v1.0.48
Compiling byteorder v1.3.2
Compiling memchr v2.2.1
Checking spin v0.5.2
Compiling semver-parser v0.7.0
Compiling log v0.4.8
Compiling typenum v1.11.2
Compiling bitflags v1.2.1
Checking slab v0.4.2
Compiling version_check v0.1.5
Compiling pkg-config v0.3.17
Compiling ryu v1.0.2
Building [=> ] 15/401: pkg-config, cc, vers...

UPDATED:

error: failed to run custom build command for libsqlite3-sys v0.17.1

Caused by:
process didn't exit successfully: /home/echedey/deltachat-core-rust/target/debug/build/libsqlite3-sys-992eb63ac34cb5ad/build-script-build (exit code: 1)
--- stdout
cargo:rerun-if-env-changed=SQLITE_MAX_VARIABLE_NUMBER
cargo:rerun-if-env-changed=SQLITE_MAX_EXPR_DEPTH
TARGET = Some("aarch64-unknown-linux-gnu")
OPT_LEVEL = Some("0")
HOST = Some("aarch64-unknown-linux-gnu")
CC_aarch64-unknown-linux-gnu = None
CC_aarch64_unknown_linux_gnu = None
HOST_CC = None
CC = None
CFLAGS_aarch64-unknown-linux-gnu = None
CFLAGS_aarch64_unknown_linux_gnu = None
HOST_CFLAGS = None
CFLAGS = None
CRATE_CC_NO_DEFAULTS = None
DEBUG = Some("true")
CARGO_CFG_TARGET_FEATURE = Some("fp,neon")
running: "cc" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-Wall" "-Wextra" "-DSQLITE_CORE" "-DSQLITE_DEFAULT_FOREIGN_KEYS=1" "-DSQLITE_ENABLE_API_ARMOR" "-DSQLITE_ENABLE_COLUMN_METADATA" "-DSQLITE_ENABLE_DBSTAT_VTAB" "-DSQLITE_ENABLE_FTS3" "-DSQLITE_ENABLE_FTS3_PARENTHESIS" "-DSQLITE_ENABLE_FTS5" "-DSQLITE_ENABLE_JSON1" "-DSQLITE_ENABLE_LOAD_EXTENSION=1" "-DSQLITE_ENABLE_MEMORY_MANAGEMENT" "-DSQLITE_ENABLE_RTREE" "-DSQLITE_ENABLE_STAT2" "-DSQLITE_ENABLE_STAT4" "-DSQLITE_SOUNDEX" "-DSQLITE_THREADSAFE=1" "-DSQLITE_USE_URI" "-DHAVE_USLEEP=1" "-DSQLITE_HAVE_ISNAN" "-o" "/home/echedey/deltachat-core-rust/target/debug/build/libsqlite3-sys-260d32a8637dc166/out/sqlite3/sqlite3.o" "-c" "sqlite3/sqlite3.c"
cargo:warning=cc1: fatal error: /usr/lib/gcc/aarch64-linux-gnu/8/include-fixed/stdc-predef.h: Mensaje erróneo
cargo:warning=compilation terminated.
exit code: 1

--- stderr

error occurred: Command "cc" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-Wall" "-Wextra" "-DSQLITE_CORE" "-DSQLITE_DEFAULT_FOREIGN_KEYS=1" "-DSQLITE_ENABLE_API_ARMOR" "-DSQLITE_ENABLE_COLUMN_METADATA" "-DSQLITE_ENABLE_DBSTAT_VTAB" "-DSQLITE_ENABLE_FTS3" "-DSQLITE_ENABLE_FTS3_PARENTHESIS" "-DSQLITE_ENABLE_FTS5" "-DSQLITE_ENABLE_JSON1" "-DSQLITE_ENABLE_LOAD_EXTENSION=1" "-DSQLITE_ENABLE_MEMORY_MANAGEMENT" "-DSQLITE_ENABLE_RTREE" "-DSQLITE_ENABLE_STAT2" "-DSQLITE_ENABLE_STAT4" "-DSQLITE_SOUNDEX" "-DSQLITE_THREADSAFE=1" "-DSQLITE_USE_URI" "-DHAVE_USLEEP=1" "-DSQLITE_HAVE_ISNAN" "-o" "/home/echedey/deltachat-core-rust/target/debug/build/libsqlite3-sys-260d32a8637dc166/out/sqlite3/sqlite3.o" "-c" "sqlite3/sqlite3.c" with args "cc" did not execute successfully (status code exit code: 1).

warning: build failed, waiting for other jobs to finish...

[link2xt]

As showed in that is written as "cfg_if". Is there any difference with it?.

It is not a bug, crate is named cfg-if, but due to rust language limitations identifier is cfg_if.

[link2xt]

@EchedeyLR could you clone https://github.com/jgallagher/rusqlite and try building it (just cargo build inside of it)? If it does not build too, we could upstream this bug (libsqlite3-sys not building) to their tracker.

[EchedelleLR]

Okay, I am going to try. I think there are different issues mixed here anyways and started with the first one which I have no idea in which sense could be related if everything is well specified.

[link2xt]

As for the original issue, try cloning and compiling https://github.com/derekjw/try_from separately from deltachat-ffi

[EchedelleLR]

@EchedeyLR could you clone https://github.com/jgallagher/rusqlite and try building it (just cargo build inside of it)? If it does not build too, we could upstream this bug (libsqlite3-sys not building) to their tracker.

https://github.com/jgallagher/rusqlite/issues/618

[EchedelleLR]

As for the original issue, try cloning and compiling https://github.com/derekjw/try_from separately from deltachat-ffi

derekjw/try_from#14

it seems both had the same issue.

[link2xt]

Maybe rust compiler/linker is simply out of memory, build processes crash and then crate can't be found. Rust could have reported this error better of course.

You probably need to cross-compile on your normal desktop machine and use resulting .a files afterwards.

[EchedelleLR]

That sounds weird. RPI3B has just 1GB ram and using microsd card is not suitable to use swap memory. Is there a way to compile using a few memory resources without crosscompiling?

[link2xt]

There is https://github.com/thepowersgang/mrustc but it seems it only supports x86.

Try reduce the number of jobs with -j 1.

[EchedelleLR]

I am trying to do it during python bindings build but I have no idea how to make it.

[link2xt]

If you are using install-python-bindings script, you can try to modify it as follows:

      subprocess.check_call([
          "cargo", "build", "-p", "deltachat_ffi", "-j", "1", "--" + target
      ])

[EchedelleLR]

The new issue is here now:

(bot) echedey@debian-pi-3b:~/deltachat-core-rust/python$ pytho
_python_bindings.py
   Compiling ryu v1.0.2
error: error writing dependencies to `/home/echedey/deltachat-core-r
/release/build/ryu-449fce45fac22562/build_script_build-449fce45fac2256
message (os error 74)

error: aborting due to previous error

error: could not compile `ryu`.

To learn more, run the command again with --verbose.
Traceback (most recent call last):
  File "install_python_bindings.py", line 24, in <module>
    "cargo", "build", "-p", "deltachat_ffi", "-j", "1", "--" + target
  File "/usr/lib/python3.7/subprocess.py", line 347, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cargo', 'build', '-p', 'delt
, '-j', '1', '--release']' returned non-zero exit status 101.

[EchedelleLR]

I will have to test a way to put swap to the rpi and try to build again as microsd are not suitable to have swapfile or swap partition.

EDITED:
Anyways, it would be great if this core could be compiled in a enviroment with limited resources similar to this situation.

[Jikstra]

Compiling rust takes time. Even on my i7 laptop i have to wait several minutes for it to complete. I don't see an option to make this easily possible on a raspi3. I think cross compiling should be the way to go, the same as we do for android phones/apple devices.

[EchedelleLR]

My common computers have similar specification but with the advantage to have 1gb more of ram and swap (until 2gb) set by default. I dont have any issues to wait but the memory requirements.

If I need to ask for someone to let me their computer to compile it is not a good solution as this use to be also common in some places where I live.

[EchedelleLR]

I added 4gb swap in a different hard drive and check some issues in dmesg related how it affects the compilation to the microsd ext4 partition (some inode errors related to high disk use). I finished doing the compilation in the hard drive where swap partition was set and this is what I get.


The previous tools including some in which i get the errors before were successfully compiled

[link2xt]

This error seems to be sort of known, looks like libc6-dev is missing: https://stackoverflow.com/questions/35822377/raspberry-pi-fatal-error-sys-cdefs-h-no-such-file-or-directory

[EchedelleLR]

Thank you. For the next time. I didnt check the repo at all but I didnt see a dependence list in expected files. This is what I check at first but as I didnt see it I just continued.

Could it be created for the next time?

[EchedelleLR]

I think it should be taken in account from the base of a boostraped debian installation or similar.

[link2xt]

Did you manage to compile it btw?

List of dependencies can sure be added, but most likely rustup + build-essential package on Debian is enough. I don't know how to make a comprehensive list, I have not reviewed the dependencies.

[EchedelleLR]

libc6-dev was already installed. I will check that issue to understand better.

[EchedelleLR]

I have had to reinstall libc6-dev and linux-libc-dev. It seems both had by default faulty files (I dont understand it) as also was referenced in the issue you pointed to me a little time before.

I hope i dont need to reinstall every build-essentials dependence ;-;

UPDATED:
395/396 by now.

[EchedelleLR]

More than 48 hours in this step.

Compiling deltachat_ffi v1.0.0-beta.23 (/mnt/GENERIC/build-env/deltachat-core-rust/deltachat-ffi)
    Building [=====================================================> ] 395/396: deltachat_ffi

I even stopped it, change "install_python_bindings" to the default conf and tried again without any change.

I think by now is not even possible to compile Delta Chat core in low resources.

[Jikstra]

Rust does a lot of crazy stuff during compiling. This takes a long time even on the ci/local system. I think the only way to do this is by cross compiling it from a stronger system for the raspi.

[link2xt]

@EchedeyLR Set lto = true to false in Cargo.toml and start again. It should be way faster.

[link2xt]

I guess we should move this lto setting to Android (and maybe iOS) build scripts instead, because other systems don't care about 10Mb that it saves.

[EchedelleLR]

I already spoke about it. No, the issue is not visible at all as far as people only look for open issues. Also, you have the projects and milestones special for developers which help to maintain up-to-date and not having to research a lot between the issues.

Also, in which case this is not a bug??. I need people to see if the problem is reproduced to be able to classify as that or not.

[r10s]

maybe, this is more sth. for the forum, as @pabzm pointed out, build works at least one some raspberry.

however, of course, i have no doubt that there are tons of build problems on different machines. but as long as there is no actionable item for the core development i would not keep it open here.

as @Jikstra pointed out, the open issues are to coordinate things among the devs - sure, there are other ways to do that, but it was decided among the devs to use the issue tracker this way.

[EchedelleLR]

I don't agree at all.

Here some users were referenced in their time from the forum to report issues here (that you and other people had to fill in the end) and is used by others users already independently of that.

It's not a "developer tool" as far I see anymore. For that you already use Projects and Milestones with the Canvan style.

[link2xt]

@EchedeyLR If you are going to work on this issue, we can reopen the issue and assign it to you.

But if you don't have time to work on it, it is not actionable, as other devs don't have Raspberry Pi around or can't reproduce the same problems.

[EchedelleLR]

I can work on it, I just needed more time than usual for it.

[EchedelleLR]

Also, I don't see acceptable to close the issue if only 2 people in completely different conditions got it work.

With a fast search about my python3-dev issue (and other header packages) I can see in stackoverflow plenty of issues about RPI using even Raspbian (not Debian in raw) and not much old with partial solutions coming from re-installing the package not working...

[EchedelleLR]

Thank you @link2xt

[r10s]

@EchedeyLR thanks for taking care!

[hpk42]

@EchedeyLR please note that we do not officially support RPI at this point -- also a lot of information is outdated here on this issue.
So if you can't get to the issue sometime soon, we'll close it and once you get to it again you can create a new issue based on trying with the current master.

[EchedelleLR]

I am going to try with the current master already.

[EchedelleLR]

It's not weekend yet which is when I can work on the issue.

[EchedelleLR]

I tried during some hours to make the system in the microsd work. The main filesystem (root) in ext4 got corrupted and I tried to repair the filesystem. I could not make it boot and I don't think to make all the configs again to use it just me.

Given that, I abandon the issue.

If other person test it, be sure that is made in the very same conditions to not get surprises when some people say that it works and it doesn't in a default installation being Raspbian or Debian.

[hpk42]

oh, sorry to hear about the filesystem corruption, that's annoying. Thanks for checking back and reporting.

…

On Sat, Jul 18, 2020 at 08:20 -0700, Echedenyan wrote: I tried during some hours to make the system in the microsd work. The main filesystem (root) in ext4 got corrupted and I tried to repair the filesystem. I could not make it boot and I don't think to make all the configs again to use it just me. Given that, I abandon the issue. If other person test it, be sure that is made in the very same conditions and to not get surprises when some people say that it works and it doesn't in a default installation being Raspbian or Debian. -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: #1189 (comment)

[link2xt]

@EchedeyLR

The main filesystem (root) in ext4 got corrupted and I tried to repair the filesystem.

This, together with random compilation errors, sounds like an underpowered RPI, check "Powering issues" section of https://ownyourbits.com/2019/02/02/whats-wrong-with-the-raspberry-pi/ and related issue raspberrypi/linux#2367

[EchedelleLR]

@EchedeyLR

The main filesystem (root) in ext4 got corrupted and I tried to repair the filesystem.

This, together with random compilation errors, sounds like an underpowered RPI, check "Powering issues" section of https://ownyourbits.com/2019/02/02/whats-wrong-with-the-raspberry-pi/ and related issue raspberrypi/linux#2367

Please, don't avoid the issue with other. The power supply fit the specification required and the issue with the microsd is related to other thing.

I didn't have such corruption issue with some 16 GB and 32 GB sd cards. The compilation was also made in external hard drive and the swap too. The issue was out of memory and I could ensure it checking htop.

And the tests adding swap in different sizes and cleaning and rebuilding the core to see if it stopped at the same step.

While the deltachat core was compiled successfully final steps involved python bindings and were not successful at all because headers were not found.

[EchedelleLR]

The issue with faulty files was something common in raspbian default installations as you can check in some random threads

[link2xt]

I am trying to build on 2020-05-27-raspios-buster-lite-armhf.img (latest Raspberry Pi OS, headless setup) now. Going to write the steps here and probably to README including the packages required once it finishes or reopen the issue if it fails.

[EchedelleLR]

@link2xt if you can, test with Debian too.

[EchedelleLR]

Generate a new one using https://salsa.debian.org/raspi-team/image-specs or get one tested here https://raspi.debian.net/tested-images/

[EchedelleLR]

Please, include specifications of your Pi model.

[link2xt]

I am using Raspberry Pi 4 Model B with 4GB RAM and 2020-05-27-raspios-buster-lite-armhf.img image.

Here is the list of steps I used to build it:

1. Install packages libssl-dev, git, virtualenv and python3-dev.
2. Create a new user with adduser to avoid changing pi user environment.
3. Install rustup from https://rustup.rs/
4. Clone https://github.com/deltachat/deltachat-core-rust.git
5. Build with cargo build -j 1 -p deltachat_ffi --release. Flag -j 1 is used to avoid running compiler in parallel and high peak memory usage. This is the most time-consuming step, maybe it is possible to speed up with parallelization, but I wanted to avoid any memory issues.
6. Create a new virtual environment for python 3: virtualenv -p /usr/bin/python3 deltachat-venv.
7. Attach virtual environment: . deltachat-venv/bin/activate.
8. Install python bindings DCC_RS_TARGET=release ./install-python-bindings.py. This step will not rebuild the core, as it is already built.
9. (optional) Install pip install pytest requests and run pytest.

[EchedelleLR]

@link2xt To be sure I checked kern.log and syslog files of the last time running were the compilation was made and didn't was anything related to under-voltage, etc.

About your specs, do you have Raspberry Pi Model 3B which is under 1GB of RAM by default?

[link2xt]

About your specs, do you have Raspberry Pi Model 3B ? which is under 1GB of RAM by default?

No, this is the only RPI I have.

[EchedelleLR]

Okay, maybe a python3-dev re-installation could have solved my issue not finding the headers correctly but as pointed before the core compiled perfectly until that step in which python-bindings installation failed.

Thank you for taking time to make a test.

[link2xt]

After you have built the core successfully in release mode, linking python bindings with the rust library is definitely possible, it is much faster and takes less memory. So not found headers is likely some package not installed or broken.

I have not tried Debian on raspberry yet, but headers should be the same as on desktop Debian, which I use every day. Going to try a few things on "official" OS first and will switch to some free OS after that, likely Debian you linked to above.