Sporadic error 502 & 503 #414

LauraErhard · 2018-10-29T14:16:30Z

Since friday we frequently get the 502 proxy error or the 503 service unavailable error. Is this because you are working on it or is it something we should be concerned about?

lgalke · 2018-10-29T21:24:20Z

It happened to me too while testing some new feature (yesterday it was quite frequently). After a couple of seconds the request succeeded. It feels like the back-end is unresponsive for a short period of time for whatever reason. @anlausch do you have an idea what causes this, and can you do something about it?

anlausch · 2018-10-30T07:22:32Z

I cannot reproduce this now. Therefore, I would suggest you penetrate the system a bit more and maybe you'll encounter certain patterns, e.g. when exactly does it happen, i.e. after which operations does the system get unstable? In the logs I can see that at some point on friday mongo was not reachable (weird), but I guess this is an exception and secondly, that for a while there where heap space issues. These I cannot trace right now, which is why I would like to ask you to help with the inspection a bit, while I am unavailable.

LauraErhard · 2018-10-30T12:36:39Z

Right now (13:29) I can't upload anything because of the 503 service unavailable error.
I tried pdf via DOI, pdf via PPN, electronic via DOI.
Update: now 5 minutes later everything works again.

LauraErhard · 2018-10-30T12:55:58Z

13:54 changing coordinates doesn't work

LauraErhard · 2018-10-30T13:03:48Z

I started a browse search (14:02) and now I get a 502 proxy error.
Update: now it works again.

lgalke · 2018-10-30T15:02:00Z

At the moment (4 PM Tuesday) 6GB of the 8GB available RAM on the server are in use. Could it be just memory problems with running more and more background jobs for four instances of our system?
Even if that is the main problem, we would need to find a way to alleviate this issue.

kleinann · 2018-10-30T16:02:13Z

17:00, Browsing in Production:
An error occurred: Http failure response for https://locdb.bib.uni-mannheim.de/locdb/getInternalSuggestionsByQueryString?query=europe&k=20: 503 Service Unavailable

Should I pass the question with the memory problems on to our IT?

kleinann · 2018-10-31T09:37:37Z

10:37h: 500 Internal Server Error while Browsing

kleinann · 2018-10-31T10:11:53Z

11:10h: Still the same error

kleinann · 2018-10-31T12:14:56Z

13:12 h, from a different computer: it works now.

anlausch · 2018-10-31T12:15:55Z

We are now using a different session store and I contacted @stweil for help. Lets see whether this improves the situation.

anlausch · 2018-10-31T12:17:23Z

@lgalke What you are assuming is in line with what I am currently thinking about the problem.

stweil · 2018-10-31T21:16:23Z

IT reconfigured the server from 8 GiB RAM + 8 GiB swap = 16 GiB memory to 24 GiB RAM + 8 GiB swap = 32 GiB memory. I expect that this will improve responsiveness because of the larger disk cache, but not solve the problem.

LauraErhard · 2018-11-05T07:20:50Z

I just had the 503 service unavailable error again (I uploaded a chapter to the demo version). It immediately worked after that, but there seems to still be an issue.

stweil · 2018-11-05T11:32:30Z

Extract from /var/log/apache2/access.log:

134.155.36.83 - - [05/Nov/2018:08:12:18 +0100] "POST /demo/saveResource HTTP/1.1" 503 762 "https://locdb.bib.uni-mannheim.de/demo-frontend/ingest" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0"

Extract from /var/log/apache2/error.log:

[Mon Nov 05 08:12:18.242821 2018] [proxy:error] [pid 21313] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:3102 (localhost) failed

Both messages occur 2 times within milliseconds. Port 3102 is handled by this process:

22562 ?        Ssl    2:47  \_ node /home/anlausch/locdb-demo/source/app.js

The process refuses connections. @anlausch, the reason for that must be searched in the code. Is there corresponding information in /home/anlausch/locdb-demo/source/combined.log? Can the process handle more than one simultaneous connections?

stweil · 2018-11-05T11:35:38Z

Older error log files also contain lots of refused connections for the ports 3000, 3101 and 3102.

stweil · 2018-11-05T11:55:54Z

I also see burst of refused connections. In the following log extract the ScanInspector tries many connections in a very short time:

[Sun Nov 04 13:55:18.013835 2018] [proxy:error] [pid 9900] [client 84.166.147.8:60170] AH00898: Error reading from remote server returned by /locdb-dev/getInternalSuggestionsByQueryString, referer: https://locdb.bib.uni-mannheim.de/extrapolate-dev/linking/ScanInspector/5bc6f8c70e466f16dff1fc21/5bc7025e0e466f16dff246c1
[Sun Nov 04 13:55:18.070894 2018] [proxy:error] [pid 12124] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:3101 (localhost) failed
[Sun Nov 04 13:55:18.070948 2018] [proxy_http:error] [pid 12124] [client 84.166.147.8:60172] AH01114: HTTP: failed to make connection to backend: localhost, referer: https://locdb.bib.uni-mannheim.de/extrapolate-dev/linking/ScanInspector/5bc6f8c70e466f16dff1fc21/5bc7025e0e466f16dff246c1
[Sun Nov 04 13:55:18.071470 2018] [proxy:error] [pid 9900] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:3101 (localhost) failed
[Sun Nov 04 13:55:18.071511 2018] [proxy_http:error] [pid 9900] [client 84.166.147.8:60170] AH01114: HTTP: failed to make connection to backend: localhost, referer: https://locdb.bib.uni-mannheim.de/extrapolate-dev/linking/ScanInspector/5bc6f8c70e466f16dff1fc21/5bc7025e0e466f16dff246c1
[Sun Nov 04 13:55:18.071992 2018] [proxy:error] [pid 10637] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:3101 (localhost) failed
[Sun Nov 04 13:55:18.072023 2018] [proxy_http:error] [pid 10637] [client 84.166.147.8:60171] AH01114: HTTP: failed to make connection to backend: localhost, referer: https://locdb.bib.uni-mannheim.de/extrapolate-dev/linking/ScanInspector/5bc6f8c70e466f16dff1fc21/5bc7025e0e466f16dff246c1
[Sun Nov 04 13:55:18.116472 2018] [proxy:error] [pid 29251] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:3101 (localhost) failed
[Sun Nov 04 13:55:18.116517 2018] [proxy_http:error] [pid 29251] [client 84.166.147.8:60174] AH01114: HTTP: failed to make connection to backend: localhost, referer: https://locdb.bib.uni-mannheim.de/extrapolate-dev/linking/ScanInspector/5bc6f8c70e466f16dff1fc21/5bc7025e0e466f16dff246c1
[Sun Nov 04 13:55:18.122283 2018] [proxy:error] [pid 3912] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:3101 (localhost) failed
[Sun Nov 04 13:55:18.122354 2018] [proxy_http:error] [pid 3912] [client 84.166.147.8:60175] AH01114: HTTP: failed to make connection to backend: localhost, referer: https://locdb.bib.uni-mannheim.de/extrapolate-dev/linking/ScanInspector/5bc6f8c70e466f16dff1fc21/5bc7025e0e466f16dff246c1
[Sun Nov 04 13:55:18.123500 2018] [proxy:error] [pid 12845] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:3101 (localhost) failed
[Sun Nov 04 13:55:18.123599 2018] [proxy_http:error] [pid 12845] [client 84.166.147.8:60176] AH01114: HTTP: failed to make connection to backend: localhost, referer: https://locdb.bib.uni-mannheim.de/extrapolate-dev/linking/ScanInspector/5bc6f8c70e466f16dff1fc21/5bc7025e0e466f16dff246c1
[Sun Nov 04 13:55:18.131569 2018] [proxy:error] [pid 12846] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:3101 (localhost) failed
[Sun Nov 04 13:55:18.131617 2018] [proxy_http:error] [pid 12846] [client 84.166.147.8:60177] AH01114: HTTP: failed to make connection to backend: localhost, referer: https://locdb.bib.uni-mannheim.de/extrapolate-dev/linking/ScanInspector/5bc6f8c70e466f16dff1fc21/5bc7025e0e466f16dff246c1
[Sun Nov 04 13:55:18.157093 2018] [proxy:error] [pid 12847] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:3101 (localhost) failed
[Sun Nov 04 13:55:18.157207 2018] [proxy_http:error] [pid 12847] [client 84.166.147.8:60178] AH01114: HTTP: failed to make connection to backend: localhost, referer: https://locdb.bib.uni-mannheim.de/extrapolate-dev/linking/ScanInspector/5bc6f8c70e466f16dff1fc21/5bc7025e0e466f16dff246c1
[Sun Nov 04 13:55:18.344617 2018] [proxy:error] [pid 8352] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:3101 (localhost) failed
[Sun Nov 04 13:55:18.344648 2018] [proxy_http:error] [pid 8352] [client 84.166.147.8:60179] AH01114: HTTP: failed to make connection to backend: localhost, referer: https://locdb.bib.uni-mannheim.de/extrapolate-dev/linking/ScanInspector/5bc6f8c70e466f16dff1fc21/5bc7025e0e466f16dff246c1
[Sun Nov 04 13:55:18.751351 2018] [proxy:error] [pid 9161] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:3101 (localhost) failed
[Sun Nov 04 13:55:18.751428 2018] [proxy_http:error] [pid 9161] [client 84.166.147.8:60180] AH01114: HTTP: failed to make connection to backend: localhost, referer: https://locdb.bib.uni-mannheim.de/extrapolate-dev/linking/ScanInspector/5bc6f8c70e466f16dff1fc21/5bc7025e0e466f16dff246c1
[Sun Nov 04 13:55:18.901096 2018] [proxy:error] [pid 3294] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:3101 (localhost) failed
[Sun Nov 04 13:55:18.901163 2018] [proxy_http:error] [pid 3294] [client 84.166.147.8:60181] AH01114: HTTP: failed to make connection to backend: localhost, referer: https://locdb.bib.uni-mannheim.de/extrapolate-dev/linking/ScanInspector/5bc6f8c70e466f16dff1fc21/5bc7025e0e466f16dff246c1
[Sun Nov 04 13:55:18.916605 2018] [proxy:error] [pid 7624] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:3101 (localhost) failed
[Sun Nov 04 13:55:18.916663 2018] [proxy_http:error] [pid 7624] [client 84.166.147.8:60182] AH01114: HTTP: failed to make connection to backend: localhost, referer: https://locdb.bib.uni-mannheim.de/extrapolate-dev/linking/ScanInspector/5bc6f8c70e466f16dff1fc21/5bc7025e0e466f16dff246c1
[Sun Nov 04 13:55:18.947858 2018] [proxy:error] [pid 3147] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:3101 (localhost) failed
[Sun Nov 04 13:55:18.947886 2018] [proxy_http:error] [pid 3147] [client 84.166.147.8:60183] AH01114: HTTP: failed to make connection to backend: localhost, referer: https://locdb.bib.uni-mannheim.de/extrapolate-dev/linking/ScanInspector/5bc6f8c70e466f16dff1fc21/5bc7025e0e466f16dff246c1
[Sun Nov 04 13:55:18.988813 2018] [proxy:error] [pid 11515] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:3101 (localhost) failed
[Sun Nov 04 13:55:18.988865 2018] [proxy_http:error] [pid 11515] [client 84.166.147.8:60184] AH01114: HTTP: failed to make connection to backend: localhost, referer: https://locdb.bib.uni-mannheim.de/extrapolate-dev/linking/ScanInspector/5bc6f8c70e466f16dff1fc21/5bc7025e0e466f16dff246c1

Are there applications which open unnecessarily many connections? The current code uses a poolSize of 10 connections, maybe that could be increased as a work around.

lgalke · 2018-11-05T14:39:48Z

The front-end triggers as few requests as possible. For instance, when saving bounding boxes, we check whether the coordinates have actually changed and only issue save requests for these ones

stweil · 2018-11-05T16:32:13Z

Status code 500 (internal server error) is related:

[Sun Nov 04 10:24:55.295789 2018] [proxy_http:error] [pid 8779] (104)Connection reset by peer: [client 84.166.145.117:50165] AH01102: error reading status line from remote server localhost:3101, referer: https://locdb.bib.uni-mannheim.de/extrapolate-dev/linking/ScanInspector/5bc6f8c70e466f16dff1fc21/5bc7025e0e466f16dff246c0

Here the JavaScript code accepted the connection, but terminated it before a full HTTP answer was sent. This can be caused by software bugs or by unexpected program exceptions like out of memory.

stweil · 2018-11-05T16:38:33Z

Even our oldest server log files already show the error pattern, so it started before Friday 2018-10-27:

/var/log/apache2/error.log.14.gz:[Mon Oct 22 09:30:54.488914 2018] [proxy:error] [pid 1361] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:3101 (localhost) failed

134.2.65.114 - - [22/Oct/2018:15:42:45 +0200] "POST /locdb-dev/bibliographicResources HTTP/1.1" 502 936 "https://locdb.bib.uni-mannheim.de/extrapolate-dev/linking/ScanInspector/5bc6f8c70e466f16dff1fc21/5bc7025e0e466f16dff246c2" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0"

LauraErhard · 2018-11-12T15:14:38Z

Is there any news on this?

anlausch · 2018-11-20T10:07:20Z

I am currently working on reproducing the issues locally. No success on this so far.

LauraErhard · 2018-11-20T14:47:46Z

If it helps, during the webmeeting I got several 502 and 503 errors.

anlausch · 2018-11-20T15:19:54Z

I prepared a modified version of our development server now, which should help to pinpoint the issue. But, I will try to make this instance fail often, so don't be confused.

LauraErhard · 2018-12-03T12:58:49Z

At the moment I have major issues. Everything I try to do gets the 503 error. Would you please check this out @lgalke @anlausch?!

stweil · 2018-12-03T13:05:59Z

It looks like the backend is down.

anlausch · 2018-12-03T13:11:43Z

There are literally no applications running anymore. I can restart them, but it is strange that they even do not appear anymore in the deployment manager. This was never the case so far. Any idea what happened?

anlausch · 2018-12-03T13:19:20Z

@stweil Maybe it is best to restart the whole server from scratch?

stweil · 2018-12-03T15:09:24Z

I restarted mongodb now.

It had failed on Saturday because of the disk full condition:

root@locdb:~# systemctl status mongodb
? mongodb.service - An object/document-oriented database
   Loaded: loaded (/lib/systemd/system/mongodb.service; enabled; vendor preset: enabled)
   Active: failed (Result: signal) since Sat 2018-12-01 00:23:48 CET; 2 days ago
     Docs: man:mongod(1)
 Main PID: 470 (code=killed, signal=ABRT)
      CPU: 2h 8min 31.299s

root@locdb:~# cat /var/log/mongodb/mongodb.log.1
[...]
Nov 22 10:03:52 locdb systemd[1]: Started An object/document-oriented database.
Dez 01 00:23:48 locdb systemd[1]: mongodb.service: Main process exited, code=killed, status=6/ABRT
Dez 01 00:23:48 locdb systemd[1]: mongodb.service: Unit entered failed state.
Dez 01 00:23:48 locdb systemd[1]: mongodb.service: Failed with result 'signal'.
[...]
2018-12-01T00:23:37.088+0100 I NETWORK  [conn17568] end connection 127.0.0.1:36498 (37 connections now open)
2018-12-01T00:23:37.743+0100 E STORAGE  [thread1] WiredTiger (28) [1543620217:743886][470:0x7f0fc6aa8700], file:WiredTiger.wt, WT_SESSION.checkpoint: /var/lib/mongodb/WiredTiger.turtle.set: handle-write: pwrite: failed to write 988 bytes at offset 0: No space left on device

anlausch · 2018-12-03T15:29:43Z

All apps should be online again.

LauraErhard · 2018-12-04T09:41:20Z

I just checked real quick and either the system is down again or still. I didn't get a chance to check yesterday so I can't say for sure.

stweil · 2018-12-04T11:36:13Z

The disk is full again, so mongodb terminated... :-(

stweil · 2018-12-04T11:58:41Z

I'm now moving the log file to another server and will restart mongodb then.

stweil · 2018-12-04T12:17:12Z

After removing the three largest log files there are now 37 GB available again. Apache2 and mongodb are up. @anlausch, please restart your services and test.

zuphilip · 2018-12-04T12:25:58Z

Yesterday, it took less than an hour to write the additional freed storage (~1GB) full (rough estimate based on the log file). In this way this will produce huge data files very easily and break the system often. We should discuss what is needed for the logging and how we can do that.

anlausch · 2018-12-04T16:55:42Z

I've now updated the logging mechanism as well as the log level. This should do the job. If not, we can make further adaptions related to writing on disk. Also, I am still working on the memory leak, which seems to be related to a module we are using for parsing marc. But this needs to be further investigated.

stweil · 2018-12-06T13:52:02Z

@anlausch, your modifications seem to work:
Now the largest log file has 0.3 GB while there are still 35 GB free disk space available.

anlausch · 2018-12-11T15:32:23Z

I just deployed an attempt to fix the leak to dev. Please test extensively.

kleinann · 2018-12-12T12:27:15Z

I'm not sure if it's the same: The 500 error while adapting bounding boxes persists:

[scan-inspector][Error] Error while deleting Entry: Http failure response for https://locdb.bib.uni-mannheim.de/locdb-dev/bibliographicEntries/5bc842f2352db22178ae8366: 500 Internal Server Erro

kleinann · 2018-12-12T12:54:41Z

Den kannte ich noch nicht:
Error while updating container: Http failure response for https://locdb.bib.uni-mannheim.de/locdb-dev/bibliographicResources: 400 Bad Request
Aus dem Log:
Object { msg: "REFERENCE TARGET SELECTED", title: "Can't see the forest for the trees?", entry_id: "5bc7238f4fb7d00dfefd9fcc", current_selected_ids: (2) […] }
logging.service.ts:140:4
Object { msg: "COMMIT PRESSED", entry_id: "5bc7238f4fb7d00dfefd9fcc", target: undefined, from_where: null }
logging.service.ts:140:4
Resource has no id. Inserting it into the backend.
Object { viewport: "PROCEEDINGS", prefix: "proceedings", data: {…}, children: undefined }
locdb.service.ts:320:6
Date setter called with null locdb.ts:534:4
Date was not valid, discarding null

kleinann · 2018-12-12T12:57:40Z

Kam gleich nochmal vor, beide Male war der Typ der Ressource Proceedings Article.

Object { msg: "REFERENCE TARGET SELECTED", title: "An architecture for the aggregation and analysis of scholarly usage data", entry_id: "5bc7238f4fb7d00dfefd9fcf", current_selected_ids: (2) […] }
logging.service.ts:140:4
Object { msg: "COMMIT PRESSED", entry_id: "5bc7238f4fb7d00dfefd9fcf", target: undefined, from_where: null }
logging.service.ts:140:4
Resource has no id. Inserting it into the backend.
Object { viewport: "PROCEEDINGS", prefix: "proceedings", data: {…}, children: undefined }
locdb.service.ts:320:6
Date setter called with null locdb.ts:534:4
Date was not valid, discarding null

anlausch · 2018-12-12T13:27:10Z

The 500 error is not related to this issue. The 400 error just tells, that there is no target for the commit given. In order to know by what these issues are caused by, we need to check, but it is at least not related to the stability problems we had. :)

stweil · 2018-12-13T15:43:49Z

Then I suggest to move @kleinann's report to a new issue.

LauraErhard · 2018-12-18T14:05:25Z

Right now I have this error again:
Could not save changes: Http failure response for https://locdb.bib.uni-mannheim.de/locdb/bibliographicResources: 502 Proxy Error

EDIT: okay, now it works again

stweil · 2018-12-18T14:18:10Z

Here is the corresponding error message on the server:

[Tue Dec 18 15:01:59.375919 2018] [proxy_http:error] [pid 24069] (20014)Internal error (specific information not available): [client 134.155.nn.nn:58911] AH01102: error reading status line from remote server localhost:3100, referer: https://locdb.bib.uni-mannheim.de/extrapolate/linking/ScanInspector/5bc090397894db015d7960b2/5bfeb2cf4ea88c74006b9427

LauraErhard · 2019-04-16T11:35:08Z

I have the 502 error again. I'm trying to upload a pdf into the production:

Send Scans failed: Object { identifier: {…}, firstpage: null, lastpage: null, file: File, _resourceType: "MONOGRAPH", uploading: false, textualPdf: false, embodimentType: "PRINT", err: {…} } Object { headers: {…}, status: 502, statusText: "Proxy Error", url: "https://locdb.bib.uni-mannheim.de/locdb/saveResource", ok: false, name: "HttpErrorResponse", message: "Http failure response for https://locdb.bib.uni-mannheim.de/locdb/saveResource: 502 Proxy Error", error: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>502 Proxy Error</title>\n</head><body>\n<h1>Proxy Error</h1>\n<p>The proxy server received an invalid\r\nresponse from an upstream server.<br />\r\nThe proxy server could not handle the request <em><a href=\"/locdb/saveResource\">POST /locdb/saveResource</a></em>.<p>\nReason: <strong>Error reading from remote server</strong></p></p>\n<hr>\n<address>Apache/2.4.25 (Debian) Server at locdb.bib.uni-mannheim.de Port 443</address>\n</body></html>\n" }

lgalke · 2019-04-16T11:38:55Z

proxy errors sometimes were connected with issues in the interface to the OCR engine. Did the API change, as you have recently updated the OCR engine @rtahseen ?

LauraErhard · 2019-04-16T12:21:08Z

I tried a different pdf and curiously enough that works. Maybe it is the pdfs fault?!
The one that didn't work: http://swb.bsz-bw.de/DB=2.1/SET=1/PRS=HOL/SHW?FRST=1&COOKIE=Us998,Pbszgast,I2017,B20728+,SY,NRecherche-DB,D2.1,E4507cacd-8a2,A,H,R193.197.31.8,FY&ACT=SRCHA

rtahseen · 2019-04-17T06:57:35Z

@lgalke Updating OCR engine can not cause this error. Earlier, such errors were caused by synchronous calling mechanism. But such errors are long gone for Automatic Reference Extractor ever since it moved to total Asynchronous calling mechanism.

By looking at the error message it looks like that it is coming from this url https://locdb.bib.uni-mannheim.de/locdb/saveResource

LauraErhard · 2019-04-23T12:21:58Z

Error 503 right now:
Object { headers: {…}, status: 503, statusText: "Service Unavailable", url: "https://locdb.bib.uni-mannheim.de/locdb/getToDo?status=OCR_PROCESSED", ok: false, name: "HttpErrorResponse", message: "Http failure response for https://locdb.bib.uni-mannheim.de/locdb/getToDo?status=OCR_PROCESSED: 503 Service Unavailable", error: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>503 Service Unavailable</title>\n</head><body>\n<h1>Service Unavailable</h1>\n<p>The server is temporarily unable to service your\nrequest due to maintenance downtime or capacity\nproblems. Please try again later.</p>\n<hr>\n<address>Apache/2.4.25 (Debian) Server at locdb.bib.uni-mannheim.de Port 443</address>\n</body></html>\n" }

LauraErhard · 2019-04-23T12:51:22Z

It's up again, @stweil rebooted the system.

LauraErhard · 2019-04-24T11:54:41Z

Our Tübinger colleages report that their system is down. They have the Error 503 as well. Can you check this out?

LauraErhard · 2019-04-24T12:08:07Z

A reboot didn't help because the system always shuts down again.

abdelqader-mohammad · 2019-04-24T14:53:12Z

Tübingen system is up now

LauraErhard · 2019-04-29T07:24:26Z

All systems (prod, dev, demo, tue) are down again.

EDIT: it works again

lgalke added back-end This issue needs to be addressed by the back-end. important labels Oct 29, 2018

lgalke mentioned this issue Oct 30, 2018

Stats page doesn't load for production #416

Closed

lgalke closed this as completed Apr 16, 2019

lgalke reopened this Apr 16, 2019

Sporadic error 502 & 503 #414

Sporadic error 502 & 503 #414

Comments

LauraErhard commented Oct 29, 2018

lgalke commented Oct 29, 2018

anlausch commented Oct 30, 2018

LauraErhard commented Oct 30, 2018

LauraErhard commented Oct 30, 2018

LauraErhard commented Oct 30, 2018

lgalke commented Oct 30, 2018 • edited Loading

kleinann commented Oct 30, 2018

kleinann commented Oct 31, 2018

kleinann commented Oct 31, 2018

kleinann commented Oct 31, 2018

anlausch commented Oct 31, 2018

anlausch commented Oct 31, 2018

stweil commented Oct 31, 2018

LauraErhard commented Nov 5, 2018

stweil commented Nov 5, 2018 • edited Loading

stweil commented Nov 5, 2018

stweil commented Nov 5, 2018

lgalke commented Nov 5, 2018

stweil commented Nov 5, 2018

stweil commented Nov 5, 2018

LauraErhard commented Nov 12, 2018

anlausch commented Nov 20, 2018

LauraErhard commented Nov 20, 2018

anlausch commented Nov 20, 2018

LauraErhard commented Dec 3, 2018

stweil commented Dec 3, 2018

anlausch commented Dec 3, 2018

anlausch commented Dec 3, 2018

stweil commented Dec 3, 2018

anlausch commented Dec 3, 2018

LauraErhard commented Dec 4, 2018

stweil commented Dec 4, 2018

stweil commented Dec 4, 2018

stweil commented Dec 4, 2018

zuphilip commented Dec 4, 2018 • edited Loading

anlausch commented Dec 4, 2018

stweil commented Dec 6, 2018

anlausch commented Dec 11, 2018

kleinann commented Dec 12, 2018

kleinann commented Dec 12, 2018

kleinann commented Dec 12, 2018

anlausch commented Dec 12, 2018

stweil commented Dec 13, 2018

LauraErhard commented Dec 18, 2018 • edited Loading

stweil commented Dec 18, 2018

LauraErhard commented Apr 16, 2019

lgalke commented Apr 16, 2019

LauraErhard commented Apr 16, 2019

rtahseen commented Apr 17, 2019

LauraErhard commented Apr 23, 2019

LauraErhard commented Apr 23, 2019

LauraErhard commented Apr 24, 2019

LauraErhard commented Apr 24, 2019

abdelqader-mohammad commented Apr 24, 2019

LauraErhard commented Apr 29, 2019 • edited Loading

lgalke commented Oct 30, 2018 •

edited

Loading

stweil commented Nov 5, 2018 •

edited

Loading

zuphilip commented Dec 4, 2018 •

edited

Loading

LauraErhard commented Dec 18, 2018 •

edited

Loading

LauraErhard commented Apr 29, 2019 •

edited

Loading