segfaults

[Abraxas0]

Hi!

I love mmproxy, it helps me a lot hiding my SSH server behind a HA proxy. Unfortunately, multiple times a day the mmproxy process crashes with a segfault.

I have no experience reporting segfaults, this is my first report :) ... so I send you the information I think you need - please ask for further information, if needed:

OS: CentOS 7.5.1804
Kernel: 3.10.0-862.el7.x86_64

Log lines:

Feb 24 03:41:28 server kernel: [10342272.180498] mmproxy[59252]: segfault at 2b3e510 ip 0000000000444934 sp 00007fff29837dd0 error 4 in mmproxy[400000+fd000]
Feb 24 03:41:39 server kernel: [10342283.488181] mmproxy[17168]: segfault at 12bc510 ip 0000000000444934 sp 00007fff93723c00 error 4 in mmproxy[400000+fd000]
Feb 24 03:43:32 server kernel: [10342395.810838] mmproxy[17218]: segfault at 135e510 ip 0000000000444934 sp 00007fff070719e0 error 4 in mmproxy[400000+fd000]
Feb 24 03:43:42 server kernel: [10342406.580925] mmproxy[17583]: segfault at 2902510 ip 0000000000444934 sp 00007ffc2cd3c780 error 4 in mmproxy[400000+fd000]

Any thoughts on that?

Thank you for that great piece of software!

[serginhohigh]

Hello. I have same problem with segmentation fault. When i try telnet mmproxy host, on host mmproxy crash and wrote in stdout - "Segmentation fault".

[majek]

I'm afraid I will need to see gdb backtrace. run mmproxy under gdb and type "bt" once it crashes.

$ gdb ./mmproxy
r --help --other-options
<when it crashes>
bt 

[Abraxas0]

I'm afraid I will need to see gdb backtrace. run mmproxy under gdb and type "bt" once it crashes.

Thank you! I will do so and commit the backtrace. Please stand by.

[Abraxas0]

Hello! I did what you have asked for and was able to reproduce a crash, but unfortunately with no backtrace (see output from gdb):

[root ~]# gdb /usr/sbin/mmproxy
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-114.el7
...
(gdb) r --verbose --mark 256 --table 100 --allowed-networks /etc/mmproxy/ips.acl --force-real-client-port -l 10.128.3.136:2222 -4 127.0.0.1:2222 -6 [::1]:2222
...
Program terminated with signal SIGSYS, Bad system call.
The program no longer exists.
(gdb)
(gdb) bt
No stack.
(gdb)

[majek]

Oh! Interesting! This might be a seccomp bug. Please disable seccomp with "--do-not-sandbox", and run "strace -C" on the running mmproxy for a while and show the output.
Basically - did we forget to whitelist some syscall?

https://github.com/cloudflare/mmproxy/blob/master/src/main.c#L564-L601

[Abraxas0]

Hello! I'm back from holyday - sorry that you had to wait for my reply.

I did what you have asked for and was not able to reproduce a crash with the --do-not-sandbox parameter, but I think I remember that I have tried this parameter before and the process also crashed after a while. Unfortunately, the process sometimes crashes after a few seconds/minutes and sometimes after hours.

This is what I did now:

[root ~]# gdb /usr/sbin/mmproxy
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-114.el7
...
(gdb) r --do-not-sandbox --verbose --mark 256 --table 100 --allowed-networks /etc/mmproxy/ips.acl --force-real-client-port -l 10.128.3.136:2222 -4 127.0.0.1:2222 -6 [::1]:2222

Find the output of "strace -C" (running for 5 minutes with multiple concurrent connections) attached:

strace.txt

Thank you for your help! Your suppor is really appriciated!

[pbennett]

Another possibility is that bad data is being passed in the proxy headers. I saw this during some initial testing where the proxy headers I created had bad data in them (I let comma delimited data slip through in the IP address). The validation of 'bad' data in the mmproxy code is fairly limited from what I saw.

[Abraxas0]

Tiny update from my side:

Despite I was not able to reproduce the behaviour manually by running mmproxy with gdb within an adequate time, the mmproxy still keeps crashing when running as a daemon and while using the --do-not-sandbox parameter for multiple times a day.

Any thoughts on that or anything I can do to sort that out?

Your help is very appriciated! Thank's in advance!

[Rar349]

I am having the same issue and can not seem to find a cause or a solution. I have tried everything mentioned here without any luck. The software is very useful but I can not continue using it for much longer if I can't get this issue nipped.

I would try to diagnose it myself but I do not really have much experience with software like this.

I would really appreciate any more possible solutions that anyone can suggest. I am still unsure if it's a problem with the software or just user error on my end.

[lindafarley]

I am in the same boat as @Rar349. I found another version that seems to work at https://github.com/path-network/go-mmproxy if it helps anyone.

[patrickbsf]

That looks fantastic and I'm a huge Go fan. Thank you @lindafarley !!! It's significantly faster as well.
What strikes me as odd is that CloudFlare is a big go user and this C code even uses a library that tries to emulate go channels/goroutines. I'm not sure why they didn't just write it in Go.

[majek]

Try 53ee896 fix

[Abraxas0]

Thank you very much for providing this fix! I've installed the new version now on a server, where we see multiple crashes every day. Will keep you up to date about the further behaviour at the end of the week.

[Abraxas0]

I downloaded and compiled the new version, including the fix, and then installed it. Unfortunately the process still crashes multiple times per hour and restarts due to systemd.

[majek]

ok, then it must be the seccomp filter

[Abraxas0]

ok, then it must be the seccomp filter

What does that mean for this issue/case? Can I do or try something?

[ghost]

ok, then it must be the seccomp filter

It cannot be the seccomp filter because it was disabled with --do-not-sandbox.

@Abraxas0 Can you please compile mmproxy with -g3 -fsanitize=address and run again (with --do-not-sandbox). It is important to get stderr and stdout

[Abraxas0]

ok, then it must be the seccomp filter

It cannot be the seccomp filter because it was disabled with --do-not-sandbox.

@Abraxas0 Can you please compile mmproxy with -g3 -fsanitize=address and run again (with --do-not-sandbox). It is important to get stderr and stdout

We do no longer use this software, as it was not stable in production and this issue has been reported two years ago. We had the urgent need for a working solution.

Anyway, thank you for your reply on this issue and your offer of support.

[ghost]

We do no longer use this software, as it was not stable in production and this issue has been reported two years ago. We had the urgent need for a working solution.

May I ask what you are using now?

[unixfox]

We do no longer use this software, as it was not stable in production and this issue has been reported two years ago. We had the urgent need for a working solution.

May I ask what you are using now?

Probably the software linked in #13 (comment)

[champtar]

You should be able to use haproxy

[Abraxas0]

We do no longer use this software, as it was not stable in production and this issue has been reported two years ago. We had the urgent need for a working solution.

May I ask what you are using now?

You, of course, may ask :)

We finally decided to use go-mmproxy (as mentioned above) behind a haproxy with proxy protocol to offer high available SFTP access to our hosting customers without missing the original IP addresses on the SFTP server (for logging and security reasons).

This setup works really fine and we never had any crashes or other issues since switching to go-mmproxy.