Self-compiled Seafile fileserver does not respond, web downloads hang

[mertemba]

Hi,
I have a problem when downloading files from the seahub web interface.
I installed Seafile on a Raspberry Pi on Arch Linux, using seafile-server (AUR) and the seafile server tarball from the official seafile download website, and deployed it on the local subnet using nginx and SQLite as described in the Arch Linux wiki entry.

Everything (syncing using the linux client, browsing the website) works, except downloading files through the seahub web interface (the file has just a few bytes):
Without an entry FILE_SERVER_ROOT in seahub_settings.py the download link redirects from http://192.168.1.10/repo/...ID.../...ID.../download/?p=/test.txt to http://192.168.1.10:8082/files/...ID.../test.txt. But when opening this link (in a browser or with curl -v --cookie ...) you get no response at all from the seafile fileserver, the browser is stuck at loading.
The port 8082 is definitively open and netstat shows seaf-server is listening on this port. I tried using nginx and FILE_SERVER_ROOT, but downloads are stuck as well, failing with a gateway timeout. In the logs/ directory there is no sign that any client has made an attempt to access a file.

Is there something wrong with my configuration, or is there a bug in the raspberry build?
Should I try to install an older version of seafile?

[mertemba]

Installing version 3.1.7 does not fix this. So this seems to be a bug in the Raspberry Pi build?

[shoeper]

Current version is 4.0.6 btw.

Information on how to deploy seafile can be found here: http://manual.seafile.com/deploy/using_sqlite.html and here http://manual.seafile.com/deploy/deploy_with_nginx.html

[mertemba]

I tried using version 4.0.6, but had no success.

Apparently the Arch Linux systemd service to start Seafile is faulty. When starting seafile manually with ./seafile.sh start and ./seahub.sh start[-fastcgi] everything works as expected.

Thanks for providing such a nice software.

[mertemba]

I have to reopen this issue, though slightly rename it, because it is still present and not due to Arch Linux stuff.
As it can be seen in the AUR some people (e.g. @Calrama) have the same issue, that a self-compiled, correctly deployed Seafile instance hangs when downloading files.
seaf-server correctly listens on port 8082 (visible via netstat) but does not respond at all when requesting a file (nginx/fastcgi does not matter). Empty (zero byte) files are returned immediately, though.
As reported version 4.0.5 did not have this bug. Any ideas?

[mertemba]

I tried a few more things, and apparently the issue is caused by libevhtp 1.2.10. When using libevhtp 1.2.9 downloads work as expected.
Is this an issue in libevhtp or was there some API change which seafile does not respect yet?

[jirutka]

I’ve tested seaf-server 4.1.5 with several versions of libevhtp:

• 1.2.0 (both upstream and haiwen’s fork), but only when this patch is applied (otherwise it can’t be compiled).

• 1.2.1 works fine, but only when compiled (and statically linked) with the bundled oniguruma!

  • If you compile it with EVHTP_BUILD_SHARED=OFF and the system-provided oniguruma (5.9.2 or 5.9.5), then seaf-server outputs this error after started:

    *** Error in `seaf-server': corrupted double-linked list: ... ***
    

  • If you compile it with EVHTP_BUILD_SHARED=ON and the system-provided oniguruma (5.9.2 or 5.9.5), then seaf-server fails when you try to download some file:

    *** stack smashing detected ***: seaf-server - terminated
    seaf-server: stack smashing attack in function <unknown> - terminated
    

• 1.2.9 works fine. However, when you compile it with EVHTP_BUILD_SHARED=ON and the system-provided oniguruma (5.9.2 or 5.9.5), then seaf-server outputs this error after started:

  *** Error in `seaf-server': corrupted double-linked list: ... ***
  

  and downloads hangs.

• 1.2.10 doesn’t work even with bundled oniguruma, downloads hangs as @jellysheep described.

I’m using Gentoo Linux, so this isn’t Arch-specific issue.

[jirutka]

It seems that it 1.2.9 with the system-provided oniguruma worked just by some weird coincidence, it doesn’t work anymore. :( This is weird.

[jurov]

I can confirm both the problem and fix (1.2.9 with included oniguruma), on gentoo + 4.1.2 seafile server. I have attached to the stuck seaf-server process with following result. It was not possible to get it to step out of epoll_wait () by connecting in any way. I forgot to save the stacktraces, only remember that Thread 3 was in malloc() and thread 1 was in libevent's event_loop. Looks like some kind of deadlock.

(gdb) info threads
  Id   Target Id         Frame 
  3    Thread 0x35cc6669700 (LWP 5845) "seaf-server" 0x0000035cc8c6920b in ?? () from /lib64/libc.so.6
* 1    Thread 0x35ccb5c2780 (LWP 5839) "seaf-server" 0x0000035cc8c5c573 in epoll_wait () from /lib64/libc.so.6

[errzey]

Augh, wish someone would have poked me about this in April.

I was helping someone else who was having a problem they attributed to libevhtp using seafile. It turned out not to be libevhtp in that case, but I started digging around this code to further assist him, and eventually made my way to pending issues. Alas I find this.

After looking around a bit, (server/access-file.c for example), there was a change in 1.2.9 that could break (technically, hanging connections) this and induce the state @jurov pasted.

I was being harassed (if you see the issue below, I'm not really joking; it got a bit hostile) about pipelined requests, and I implemented a fix to do just that.

But it touched some logic that the seafile code uses, and I am completely sorry & embarrassed by saying it.

"you call resume AFTER you have sent the reply (not BEFORE)." or in short: treat it like pending data, then hit the resume button.

Long story short: pipelined request processing is hard. (https://github.com/ellzey/libevhtp/issues/86 <- it's like 432423 pages long).

But it came down to this:

• When a request is paused, both EV_READ AND EV_WRITE are disabled.
• When a request is resumed, if there is data waiting to be sent to the client, a "wait for the write to finish" flag is set, and ONLY EV_WRITE is enabled.
• Once that part has been fully written, the "wait for write" flag is unset, and EV_READ is enabled again.

I would be happy to go through all this code and make sure that the order is correct.

[freeplant]

@ellzey Thank you very much for the information. We will give new version of libevhtp a fully test in the next 2 weeks.

[errzey]

So here is the problem, again:

When EV_WRITE is enabled, and there is no more data that is written, then nothing happens. This was fixed in libevent2.1x. with bufferevent_trigger which is like event_active but you can trigger specific data callbacks; on top of that it becomes deferred, which is good in cases like this.

I'm trying to figure out how to emulate the function in 2.0.

[errzey]

I haven't left yet, I just pushed some testing code that gets around using libevent 2.1

https://github.com/ellzey/libevhtp/tree/feature/resume_fixes

Prior to this, the following logic for pause/resume was:

1. When a request is paused, both EV_READ AND EV_WRITE are disabled.
2. When a request is resumed, if there is data waiting to be sent to the
   client, a "wait for the write to finish" flag is set, and ONLY EV_WRITE
   is enabled.
3. Once that part has been fully written, the "wait for write" flag is
   unset, and EV_READ is enabled again.

This can run into issues:

When EV_WRITE is enabled, and there is no more data that is written,
then nothing happens. This was fixed in libevent2.1x. with
bufferevent_trigger which is like event_active but you can trigger
specific data callbacks; on top of that it becomes deferred, which is
good in cases like this.

The new proposed logic is as follows:
1. Same
2. When a request is resumed, if data is awaiting, instead of
   just calling bufferevent_enable(), we also call event_active
   on the underlying bufferevent's write callback.
3. When a request is resumed and ready to read, a new event
   has been created: `resume_read_ev` which (in the next
   loop) will execute the readcb once it's activated.

Other:
Added debugging stuff everywhere if enabled.
Added a clearer example of pause/resume in examples/test_pause_resume.c

Much testing from my side, along with better documentation of course.

[niol]

Please see https://github.com/haiwen/seafile-server/issues/67#issuecomment-337904800 , it seems to work.