Promtail 2.7.0 won't start on OpenWRT 22.03 x86/64.

[bodyscape]

Describe the bug
Promtail won't start on OpenWRT 22.03 x86/64.
When i try to run promtail with this cmd : ./promtail-linux-amd64
I have this error : -ash: ./promtail-linux-amd64: not found
Really weird because the file is there and have the execute right like the loki-linux-amd64.

Expected behavior
I think it's suppose to start because Loki, Logcli and loki-canary work with the same version 2.7.0.

Environment:
OpenWrt 22.03.2 r19803-9a599fee93 / LuCI openwrt-22.03 branch git-22.304.65171-ec905e6
Linux Kernel 5.10.146
Promtail 2.7.0

Screenshots, Promtail config, or terminal output

[dannykopping]

@bodyscape can you please send a directory listing of this directory?
By definition this can't be a promtail issue because the promtail binary is not being found, but I'll see if I can help.

[bodyscape]

@dannykopping

I think your right, i just don't understand. ;) Thanks for the help.

[dannykopping]

Only thing I can think of here is to run strace:

$ strace -f -o /tmp/strace ./promtail-linux-amd64

Post the contents of /tmp/strace (bearing in mind there may be sensitive details), and I'll see if anything stands out.

[bodyscape]

No strace file is created if a run promtail or loki.

[dannykopping]

Try run echo $$ to get the pid of this pseudo-terminal (A).
Then, in another shell (B), run strace -f -o /tmp/strace -p <the pid>. Once you have that running, run ./promtail-linux-amd64 in (A).

[bodyscape]

Sorry the file was there the first time.. my bad!!!

where i run strace -f -o /tmp/strace ./promtail-linux-amd64 :
5629 execve("./promtail-linux-amd64", ["./promtail-linux-amd64"], 0x7ffc2a674e18 /* 15 vars */) = -1 ENOENT (No such file or directory)
5629 +++ exited with 1 +++

When i run the 2 terminal like you said :
3590 restart_syscall(<... resuming interrupted read ...>) = 1
3590 read(0, ".", 1) = 1
3590 writev(1, [{iov_base=".", iov_len=1}, {iov_base=NULL, iov_len=0}], 2) = 1
3590 poll([{fd=0, events=POLLIN}], 1, -1) = 1 ([{fd=0, revents=POLLIN}])
3590 read(0, "/", 1) = 1
3590 writev(1, [{iov_base="/", iov_len=1}, {iov_base=NULL, iov_len=0}], 2) = 1
3590 poll([{fd=0, events=POLLIN}], 1, -1) = 1 ([{fd=0, revents=POLLIN}])
3590 read(0, "p", 1) = 1
3590 writev(1, [{iov_base="p", iov_len=1}, {iov_base=NULL, iov_len=0}], 2) = 1
3590 poll([{fd=0, events=POLLIN}], 1, -1) = 1 ([{fd=0, revents=POLLIN}])
3590 read(0, "r", 1) = 1
3590 writev(1, [{iov_base="r", iov_len=1}, {iov_base=NULL, iov_len=0}], 2) = 1
3590 poll([{fd=0, events=POLLIN}], 1, -1) = 1 ([{fd=0, revents=POLLIN}])
3590 read(0, "\t", 1) = 1
3590 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f867820b000
3590 open("./", O_RDONLY|O_LARGEFILE|O_CLOEXEC|O_DIRECTORY) = 3
3590 fcntl(3, F_SETFD, FD_CLOEXEC) = 0
3590 getdents64(3, 0x7f867820e6f8 /* 6 entries /, 2048) = 208
3590 stat("./promtail-linux-amd64", {st_mode=S_IFREG|0777, st_size=84666752, ...}) = 0
3590 getdents64(3, 0x7f867820e6f8 / 0 entries /, 2048) = 0
3590 close(3) = 0
3590 munmap(0x7f867820b000, 4096) = 0
3590 writev(1, [{iov_base="\r\33]0;root@Security: /mnt/sda3/pr"..., iov_len=99}, {iov_base=NULL, iov_len=0}], 2) = 99
3590 poll([{fd=0, events=POLLIN}], 1, -1) = 1 ([{fd=0, revents=POLLIN}])
3590 read(0, "\n", 1) = 1
3590 writev(1, [{iov_base="", iov_len=0}, {iov_base="\n", iov_len=1}], 2) = 1
3590 ioctl(0, SNDCTL_TMR_START or TCSETS, {B38400 opost isig icanon echo ...}) = 0
3590 rt_sigprocmask(SIG_BLOCK, [RTMIN RT_1 RT_2], [], 8) = 0
3590 rt_sigprocmask(SIG_BLOCK, [], [KILL STOP RTMIN RT_1 RT_2], 8) = 0
3590 fork() = 5633
3590 rt_sigprocmask(SIG_SETMASK, [KILL STOP RTMIN RT_1 RT_2], NULL, 8) = 0
3590 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
3590 setpgid(5633, 5633) = 0
3590 wait4(-1, <unfinished ...>
5633 gettid() = 5633
5633 rt_sigprocmask(SIG_SETMASK, [KILL STOP RTMIN RT_1 RT_2], NULL, 8) = 0
5633 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
5633 getpid() = 5633
5633 setpgid(0, 5633) = 0
5633 ioctl(10, TIOCSPGRP, [5633]) = 0
5633 rt_sigaction(SIGTSTP, {sa_handler=SIG_DFL, sa_mask=[RTMIN RT_1 RT_2], sa_flags=SA_RESTORER, sa_restorer=0x7f867826ceba}, NULL, 8) = 0
5633 rt_sigaction(SIGTTOU, NULL, {sa_handler=SIG_IGN, sa_mask=[], sa_flags=0}, 8) = 0
5633 rt_sigaction(SIGTTOU, {sa_handler=SIG_DFL, sa_mask=[RTMIN RT_1 RT_2], sa_flags=SA_RESTORER, sa_restorer=0x7f867826ceba}, NULL, 8) = 0
5633 rt_sigaction(SIGINT, {sa_handler=SIG_DFL, sa_mask=[RTMIN RT_1 RT_2], sa_flags=SA_RESTORER, sa_restorer=0x7f867826ceba}, NULL, 8) = 0
5633 rt_sigaction(SIGTERM, {sa_handler=SIG_DFL, sa_mask=[RTMIN RT_1 RT_2], sa_flags=SA_RESTORER, sa_restorer=0x7f867826ceba}, NULL, 8) = 0
5633 rt_sigaction(SIGQUIT, {sa_handler=SIG_DFL, sa_mask=[RTMIN RT_1 RT_2], sa_flags=SA_RESTORER, sa_restorer=0x7f867826ceba}, NULL, 8) = 0
5633 execve("./promtail-linux-amd64", ["./promtail-linux-amd64"], 0x7f867829c190 / 15 vars */) = -1 ENOENT (No such file or directory)
5633 writev(2, [{iov_base="-ash: ", iov_len=6}, {iov_base=NULL, iov_len=0}], 2) = 6
5633 writev(2, [{iov_base="./promtail-linux-amd64: not foun"..., iov_len=33}, {iov_base=NULL, iov_len=0}], 2) = 33
5633 writev(2, [{iov_base="", iov_len=0}, {iov_base="\n", iov_len=1}], 2) = 1
5633 exit_group(127) = ?
5633 +++ exited with 127 +++
3590 <... wait4 resumed>[{WIFEXITED(s) && WEXITSTATUS(s) == 127}], WSTOPPED, NULL) = 5633
3590 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=5633, si_uid=0, si_status=127, si_utime=0, si_stime=0} ---
3590 rt_sigreturn({mask=[]}) = 5633
3590 wait4(-1, 0x7ffd0546d28c, WNOHANG|WSTOPPED, NULL) = -1 ECHILD (No child process)
3590 ioctl(10, TIOCSPGRP, [3590]) = 0
3590 ioctl(0, TCGETS, {B38400 opost isig icanon echo ...}) = 0
3590 ioctl(0, SNDCTL_TMR_START or TCSETS, {B38400 opost -isig -icanon -echo ...}) = 0
3590 ioctl(0, TIOCGWINSZ, {ws_row=38, ws_col=133, ws_xpixel=0, ws_ypixel=0}) = 0
3590 geteuid() = 0
3590 open("/etc/passwd", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
3590 fcntl(3, F_SETFD, FD_CLOEXEC) = 0
3590 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f867820b000
3590 fcntl(3, F_SETFD, FD_CLOEXEC) = 0
3590 read(3, "root:x:0:0:root:/root:/bin/ash\nd"..., 1024) = 374
3590 lseek(3, -343, SEEK_CUR) = 31
3590 close(3) = 0
3590 munmap(0x7f867820b000, 4096) = 0
3590 uname({sysname="Linux", nodename="Security", ...}) = 0
3590 getcwd("/mnt/sda3/promtail", 192) = 19
3590 uname({sysname="Linux", nodename="Security", ...}) = 0
3590 geteuid() = 0
3590 writev(1, [{iov_base="\33]0;root@Security: /mnt/sda3/pro"..., iov_len=72}, {iov_base=NULL, iov_len=0}], 2) = 72
3590 poll([{fd=0, events=POLLIN}], 1, -1

[dannykopping]

Thanks for this.
OK, I suspect a problem with your disk perhaps. Try executing promtail with the full (not relative) path?
Try deleting the file and downloading it again
Try checking the sha256 against https://github.com/grafana/loki/releases/download/v2.7.0/SHA256SUMS

If none of those work, then I don't really have many other ideas right now

[bodyscape]

I already try the full path and downloading again serveral time. Also test other version. don't work.

I try to validate the checksum like you said, they are the same.

Do you know if there are minimum requirement to run it?

[dannykopping]

Not really, just a couple hundred MB of RAM.
In any case, it wouldn't manifest in this way.

This looks like an OS/hardware issue, not one with Promtail, so unfortunately I think I'll have to close this issue - but please do let us know if you find the issue! If you find evidence that it is a Promtail issue, we can certainly reopen this.

Best of luck @bodyscape

[bodyscape]

Thanks for your help. :) @dannykopping

[bodyscape]

@dannykopping

Anyway idea for the dynamic link??

If a copy the files from a other linux promtail start but the VM don't boot up anymore. ;)

[dannykopping]

Oh interesting... that seems odd. Nice catch!
@trevorwhitney do you know anything about this? Did we change the way these binaries are built for 2.7?

[bodyscape]

@dannykopping

i build from the source and i add this flag like the upper line. Now it's work flawless on my OpenWRT. :)

[vlad-diachenko]

Hey @bodyscape . it looks like we already had such an issue (here) ;)
I will figure out a few questions about the binaries build process to see if we can build a static binary with enabled CGO... I will keep you updated.

[trevorwhitney]

@dannykopping we did change the build for loki a bit, using go directly instead of gox, and adding -trimpath flags, and maybe the flag stuff carried over for promtail as well?

[vlad-diachenko]

@dannykopping we did change the build for loki a bit, using go directly instead of gox, and adding -trimpath flags, and maybe the flag stuff carried over for promtail as well?

hey @trevorwhitney . we still use gox to build binaries that are published to github. https://github.com/grafana/loki/blob/main/Makefile#L260

[vlad-diachenko]

hey @bodyscape . do you have any updates? did this comment help to you ?
Also, taking into account this comment, it looks like we can not build fully static binary.

[bodyscape]

Hey @vlad-diachenko. It's ok, like i said. I made my custom build with the config i change. it's work great since.

Thanks for your help.