n8n: fails to start on aarch64

[purcell]

Describe the bug

Wanted to try out n8n on a Raspberry Pi 4 running latest nixos-unstable, so I enabled it with no specific startup configuration, and it exits with the following error:

Apr 17 17:34:28 media n8n[447342]: #
Apr 17 17:34:28 media n8n[447342]: # Fatal error in , line 0
Apr 17 17:34:28 media n8n[447342]: # Check failed: reservation_.SetPermissions(protect_start, protect_size, permission).
Apr 17 17:34:28 media n8n[447342]: #
Apr 17 17:34:28 media n8n[447342]: #
Apr 17 17:34:28 media n8n[447342]: #
Apr 17 17:34:28 media n8n[447342]: #FailureMessage Object: 0x7fc9955fd8
Apr 17 17:34:28 media n8n[447342]:  1: 0x9f42d8  [/nix/store/way40gr84vrk7plfm1kq9zabdjmc0jnm-nodejs-14.16.1/bin/node]
Apr 17 17:34:28 media n8n[447342]:  2: 0x1469b34 V8_Fatal(char const*, ...) [/nix/store/way40gr84vrk7plfm1kq9zabdjmc0jnm-nodejs-14.16.1/bin/node]
Apr 17 17:34:28 media n8n[447342]:  3: 0xd03d04  [/nix/store/way40gr84vrk7plfm1kq9zabdjmc0jnm-nodejs-14.16.1/bin/node]
Apr 17 17:34:28 media n8n[447342]:  4: 0xd26cd0 v8::internal::PagedSpace::SetReadAndExecutable() [/nix/store/way40gr84vrk7plfm1kq9zabdjmc0jnm-nodejs-14.16.1/bin/node]
Apr 17 17:34:28 media n8n[447342]:  5: 0xc55664 v8::internal::Isolate::Init(v8::internal::ReadOnlyDeserializer*, v8::internal::StartupDeserializer*) [/nix/store/way40gr84vrk7plfm1kq9zabdjmc0jnm-nodejs-14>
Apr 17 17:34:28 media n8n[447342]:  6: 0x105997c  [/nix/store/way40gr84vrk7plfm1kq9zabdjmc0jnm-nodejs-14.16.1/bin/node]
Apr 17 17:34:28 media n8n[447342]:  7: 0xb2bf3c v8::Isolate::Initialize(v8::Isolate*, v8::Isolate::CreateParams const&) [/nix/store/way40gr84vrk7plfm1kq9zabdjmc0jnm-nodejs-14.16.1/bin/node]
Apr 17 17:34:28 media n8n[447342]:  8: 0x9c6e84 node::NodeMainInstance::NodeMainInstance(v8::Isolate::CreateParams*, uv_loop_s*, node::MultiIsolatePlatform*, std::vector<std::__cxx11::basic_string<char, >
Apr 17 17:34:28 media n8n[447342]:  9: 0x955088 node::Start(int, char**) [/nix/store/way40gr84vrk7plfm1kq9zabdjmc0jnm-nodejs-14.16.1/bin/node]
Apr 17 17:34:28 media n8n[447342]: 10: 0x7fa2ab97f0 __libc_start_main [/nix/store/al61kv9rp0rriqdfczyby59an2ax7xyz-glibc-2.32-40/lib/libc.so.6]
Apr 17 17:34:28 media n8n[447342]: 11: 0x8ebe78 _start [/nix/store/way40gr84vrk7plfm1kq9zabdjmc0jnm-nodejs-14.16.1/bin/node]
Apr 17 17:34:28 media systemd[1]: n8n.service: Main process exited, code=dumped, status=5/TRAP

I don't see any corresponding issues in github.

To Reproduce
Steps to reproduce the behavior:

1. aarch64 machine, nixos-unstable
2. services.n8n.enable = true
3. nixos-rebuild switch
4. journalctl -u n8n

Expected behavior

Successful startup and accessible by web interface.

Screenshots

Notify maintainers

@freezeboy, @arjunv27

Metadata

• system: "aarch64-linux"
• host os: Linux 5.4.79, NixOS, 21.05pre282432.311ceed827f (Okapi)
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.3.10
• channels(root): "nixos-21.05pre282432.311ceed827f"
• channels(steve): ""
• channels(nixos): ""
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
 - n8n
# a list of nixos modules affected by the problem
module:
 - n8n

[freezeboy]

Hi, I just tried the binary on my machine from master, looks ok, but it is not an aarch64.
I will look into it, but I won't be able to verify the behavior on this arch

[freezeboy]

@purcell Can you please check if a basic call to node or npm on the raspberry is also crashing ?

[ulrikstrid]

I can second that enabling the n8n service on raspberry pi 4 fails.

It seems to fail for me on other machines as well but with a slightly different error message. I don't have the errors at hand but can try to post when I have my computer.

[freezeboy]

In fact I run the nixosTests.n8n locally and it also fails inside virtualization, with an error that looks similar too :/
I'm not sure what counter measures we can take, I also tried to update the software but the behavior is the same

[ulrikstrid]

On my non-arm machine I recall that it looks like a permission error when starting node.
I can run node fine when just running it plain.

Edit, here's the output on non-arm64:

Started N8N service.
#
# Fatal error in , line 0
# Check failed: reservation_.SetPermissions(protect_start, protect_size, permission).
#
#
#
#FailureMessage Object: 0x7ffc69bb8e50
1: 0x9c69d5  [/nix/store/y9ay04l5mfm255r296vhcjbxjqkjxp39-nodejs-14.16.1/bin/node]
2: 0x15210c9 V8_Fatal(char const*, ...) [/nix/store/y9ay04l5mfm255r296vhcjbxjqkjxp39-nodejs-14.16.1/bin/node]
3: 0xd1cac3  [/nix/store/y9ay04l5mfm255r296vhcjbxjqkjxp39-nodejs-14.16.1/bin/node]
4: 0xd4a4a8 v8::internal::PagedSpace::SetReadAndExecutable() [/nix/store/y9ay04l5mfm255r296vhcjbxjqkjxp39-nodejs-14.16.1/bin/node]
5: 0xc5bad2 v8::internal::Isolate::Init(v8::internal::ReadOnlyDeserializer*, v8::internal::StartupDeserializer*) [/nix/store/y9ay04l5mfm255r296vhcjbxjqkjxp39-nodejs-14.16.1/bin/node]
6: 0x10b98f3  [/nix/store/y9ay04l5mfm255r296vhcjbxjqkjxp39-nodejs-14.16.1/bin/node]
7: 0xb151a7 v8::Isolate::Initialize(v8::Isolate*, v8::Isolate::CreateParams const&) [/nix/store/y9ay04l5mfm255r296vhcjbxjqkjxp39-nodejs-14.16.1/bin/node]
8: 0x99154a node::NodeMainInstance::NodeMainInstance(v8::Isolate::CreateParams*, uv_loop_s*, node::MultiIsolatePlatform*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx>basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<unsigned long, std::allocator<unsigned long> > const*) [/nix/store/y9ay04l5mfm255r296vhcjbxjqkjxp39-nodejs-14.16.1/bin/node]
9: 0x90d4d4 node::Start(int, char**) [/nix/store/y9ay04l5mfm255r296vhcjbxjqkjxp39-nodejs-14.16.1/bin/node]
10: 0x7fc37d320ded __libc_start_main [/nix/store/1jn6apz0fa9h9x7rl3v6vwiymwnjznwv-glibc-2.32-40/lib/libc.so.6]
11: 0x898daa _start [/nix/store/y9ay04l5mfm255r296vhcjbxjqkjxp39-nodejs-14.16.1/bin/node]
n8n.service: Main process exited, code=dumped, status=4/ILL

[ulrikstrid]

A guess is that this folder is owned by the wrong user.

[freezeboy]

I tried the n8n derivation from unstable on my 20.09 system and it looks good, even as a service. This module has some hardening features enabled in the systemd unit file. Maybe the systemd update from 20.09 -> unstable could explain the change? I don't have a running system with unstable kernel/systemd

[freezeboy]

A guess is that this folder is owned by the wrong user.

But this folder is generated by systemd on startup

[purcell]

A guess is that this folder is owned by the wrong user.

It's owned by root. Does that count as the wrong user?

[purcell]

Actually, only the symlink is owned by root. Here are the corresponding directories:

steve@media> ls -la /var/lib/n8n
lrwxrwxrwx 1 root root 11 Apr 17 17:34 /var/lib/n8n -> private/n8n
steve@media> sudo ls -lad /var/lib/private/n8n
drwxr-xr-x 2 64383 64383 4096 Apr 17 17:34 /var/lib/private/n8n

So it looks like /var/lib/private/n8n is assigned to the DynamicUser created by systemd.

But in any case, it looks like the error is related to nodejs/node#37061, supposedly fixed in Node 15.9.0

[purcell]

Scratch that, I think the error message for that issue is a bit different. There's talk here of fixing this specific error via the attributes of the node executable, e.g. setfattr -n user.pax.flags -v "mr" $(find $NVM_DIR -type f -iname "node" -o -iname "npm" -o -iname "npx").

[freezeboy]

Scratch that, I think the error message for that issue is a bit different. There's talk here of fixing this specific error via the attributes of the node executable, e.g. setfattr -n user.pax.flags -v "mr" $(find $NVM_DIR -type f -iname "node" -o -iname "npm" -o -iname "npx").

I am not sure is matches the nix store requirements

[purcell]

Yeah, that's definitely not a fix, just noting related info I found.

[purcell]

I'll take the liberty of CC-ing the nodejs package maintainers here in case the issue is obvious to them: @goibhniu @gilligan (hi!) @cko @marsam

[baloo]

pretty sure it's because of the:

MemoryDenyWriteExecute = "yes"

in systemd' service configuration. This is incompatible with nodejs afaik, because its JIT relies on the fact it can rewrite the program's code dynamically.

[purcell]

This is incompatible with nodejs afaik, because its JIT relies on the fact it can rewrite the program's code dynamically.

Indeed! I see mentions of this elsewhere, e.g. here and here. I'll send a PR.