Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unknown error without any info/log to troubleshoot #151

Open
arkanoid87 opened this issue Oct 9, 2019 · 22 comments
Open

Unknown error without any info/log to troubleshoot #151

arkanoid87 opened this issue Oct 9, 2019 · 22 comments

Comments

@arkanoid87
Copy link

Do you want to request a feature or report a bug?
bug

What is the current behavior?
Unknown error without any info

If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem.
client > TLS terminator reverse proxy (traefik) > nextcloud (+ app) + documentserver
using traefik:latest, nextcloud:latest and onlyoffice/documentserver:latest in a docker-compose stack.
Stacks seems correctly configured: I successfully set all the urls (even http internal ones) in nextcloud app config and both nextcloud and documentserver public https endpoints are reachable.
Problem: open ANY document (tested newly created one and the xlsx you provide as example), tab opens, onlyoffice seems to load, then a misterious "unknown error" popup.

What is the expected behavior?
Documents opens correctly in new tab

Did this work in previous versions of DocumentServer?
Tested 5.3.0.243, same problem there

DocumentServer version:
5.4.1.39

Operating System:
Ubuntu server 18.04

Browser version:
Firefox 69.0.1

I've been searching for hints on how to debug the problem for hours unsuccessfully.
Been loooking into:

  • browser js console
  • traefik logs
  • nextcloud logs
  • documentserver logs

Also tried:

  • disable CSP
  • tail -f /var/log/onlyoffice/documentserver/**/*.log (no output here when triggering the problem)
  • tail -f /var/www/html/data/nextcloud.log (no ouput here when triggering the problem)

More info:

  • browser dev tools shows no network errors (code 200 everywhere)

2019-10-10-002835_812x602_scrot

It's issue ONLYOFFICE/DocumentServer#666 for a reason

@arkanoid87
Copy link
Author

arkanoid87 commented Oct 10, 2019

By manually analysing the js stacktrace I ended up nearby this failed XMLHttpRequest

"wss://office.mydomain.com/cache/files/4009520959/Editor.bin/Editor.bin?md5=vyDJoVrnjfh3o02BppqNsA&expires=1573258415&disposition=attachment&ooname=output.bin"

EDIT:
testing websocket connection manually from remote host with websocat tool returns valid connection:

websocat -vvv wss://office.mydomain.com/5.4.1-39//doc/3555953290/c/259/vtqjoun2/websocket
...
[INFO websocat::ws_client_peer] Connected to ws
[DEBUG websocat::ws_peer] incoming text
...
["{"type":"license","license":{"type":3,"light":false,"mode":0,"rights":1,"buildVersion":"5.4.1","buildNumber":39,"branding":false,"customization":false,"plugins":false}}"]

also
websocat -vvv wss://office.mydomain.com/5.4.1-39//spellchecker/doc/3555953290/c/497/ebs0u2ik/websocket
works correctly

@arkanoid87
Copy link
Author

arkanoid87 commented Oct 10, 2019

I've tried removing TLS termination on reverse proxy but the problem is still there even with 100% HTTP

I've increased to 'loglevel' => 0 and this is the output of data/nextcloud.log when I trigger the problem

{"reqId":"2GkFSnxWzA3EFNtCoJzt","level":0,"time":"2019-10-10T02:16:49+00:00","remoteAddr":"2.238.151.49","user":"myuser","app":"onlyoffice","method":"GET","url":"/apps/onlyoffice/320?filePath=%2FExample%20Spreadsheet%20Title.xlsx","message":"Open: 320 /Example Spreadsheet Title.xlsx","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:69.0) Gecko/20100101 Firefox/69.0","version":"17.0.0.9"}
{"reqId":"p0Km3mT3Fo9uC4Hvm4ZC","level":0,"time":"2019-10-10T02:16:51+00:00","remoteAddr":"2.238.151.49","user":"myuser","app":"onlyoffice","method":"GET","url":"/apps/onlyoffice/ajax/config/320?filePath=%2FExample%20Spreadsheet%20Title.xlsx","message":"Config is generated for: 320 with key ocz4c80wietg_320_1570648071","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:69.0) Gecko/20100101 Firefox/69.0","version":"17.0.0.9"}
{"reqId":"c3vUnoLQzpWmO4MIz1wx","level":0,"time":"2019-10-10T02:16:53+00:00","remoteAddr":"172.21.0.4","user":"--","app":"onlyoffice","method":"POST","url":"/apps/onlyoffice/track?doc=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJmaWxlSWQiOjMyMCwib3duZXJJZCI6ImphY2siLCJzaGFyZVRva2VuIjpudWxsLCJhY3Rpb24iOiJ0cmFjayJ9.6tqJL30rrmFIxdgbAB0YcDZuDngzZ1OaZheevtY9rgo","message":"Track: 320 status 1","userAgent":"--","version":"17.0.0.9"}
{"reqId":"c3vUnoLQzpWmO4MIz1wx","level":0,"time":"2019-10-10T02:16:53+00:00","remoteAddr":"172.21.0.4","user":"--","app":"onlyoffice","method":"POST","url":"/apps/onlyoffice/track?doc=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJmaWxlSWQiOjMyMCwib3duZXJJZCI6ImphY2siLCJzaGFyZVRva2VuIjpudWxsLCJhY3Rpb24iOiJ0cmFjayJ9.6tqJL30rrmFIxdgbAB0YcDZuDngzZ1OaZheevtY9rgo","message":"Track: 320 status 1 result 0","userAgent":"--","version":"17.0.0.9"}

@ShockwaveNN ShockwaveNN transferred this issue from ONLYOFFICE/DocumentServer Oct 10, 2019
@raph2i
Copy link

raph2i commented Oct 16, 2019

Hey guys, same issue here. Traefik 2 - nextcloud-17-apache - documentserver

@alexanderonlyoffice
Copy link

Hello @arkanoid87, it looks likes there is some problem with the functioning of the Document Server.

Please check the Document Server by entering the address https://documentserver_address/healthcheck/ and send us the response you get in browser.

Enable extending logging for the Document Server: open /etc/onlyoffice/documentserver/log4js/production.json and replace the 'WARN' value with 'DEBUG' of the "level" parameter. Then restart all services of the Document Server with a command supervisorctl restart all. Then open ONLYOFFICE integration app settings page in Nextcloud and click on the 'Save' button. check the logs after that.

@samuel-p
Copy link

I have the same issue since I upgraded Traefik to v2. /healthcheck/ returns true.

@derdrave
Copy link

derdrave commented Nov 4, 2019

Same here with traefik2(.0.4)

i changed logging to debug and did two things: First i went to the settings-page and saved my settings again and second i tried to open a document.. where the referenced error appears.

the nginx-log from location "/var/log/onlyoffice/documentserver/nginx.error.log" shows these errors:

2019/11/04 10:09:05 [error] 710#710: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.0.2, server: , request: "GET /welcome/ HTTP/1.1", upstream: "http://127.0.0.1:8000/welcome/", host: "office.yourserver.com"
2019/11/04 10:09:05 [error] 710#710: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.0.2, server: , request: "GET /welcome/ HTTP/1.1", upstream: "http://127.0.0.1:8000/welcome/", host: "office.yourserver.com"
2019/11/04 10:09:05 [error] 710#710: *1 no live upstreams while connecting to upstream, client: 172.18.0.2, server: , request: "GET /welcome/ HTTP/1.1", upstream: "http://docservice/welcome/", host: "office.yourserver.com"
2019/11/04 10:09:05 [error] 710#710: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.0.2, server: , request: "GET /welcome/ HTTP/1.1", upstream: "http://127.0.0.1:8000/welcome/", host: "office.yourserver.com"
2019/11/04 10:09:05 [error] 710#710: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.0.2, server: , request: "GET /welcome/ HTTP/1.1", upstream: "http://127.0.0.1:8000/welcome/", host: "office.yourserver.com"
2019/11/04 10:09:06 [error] 710#710: *1 no live upstreams while connecting to upstream, client: 172.18.0.2, server: , request: "GET /welcome/ HTTP/1.1", upstream: "http://docservice/welcome/", host: "office.yourserver.com"
2019/11/04 10:09:06 [error] 710#710: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.0.2, server: , request: "GET /welcome/ HTTP/1.1", upstream: "http://127.0.0.1:8000/welcome/", host: "office.yourserver.com"
2019/11/04 10:09:06 [error] 710#710: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.0.2, server: , request: "GET /welcome/ HTTP/1.1", upstream: "http://127.0.0.1:8000/welcome/", host: "office.yourserver.com"
2019/11/04 10:09:06 [error] 710#710: *1 no live upstreams while connecting to upstream, client: 172.18.0.2, server: , request: "GET /welcome/ HTTP/1.1", upstream: "http://docservice/welcome/", host: "office.yourserver.com"

--> client ip (client: 172.18.0.2) is internal ip of my traefik.. i think it should be either the nextcloud-ip or my real-client-ip?

when i set up onlyoffice and nextcloud dockers with an haproxy docker (official) i have no problems with my setups..

@derdrave
Copy link

derdrave commented Nov 4, 2019

i found a temporary solution by reconfiguring nginx inside the docker:

in file: /etc/nginx/includes/http-common.conf
i commented out this line (bottom of the file):

proxy_set_header Connection $proxy_connection; 
-->
# proxy_set_header Connection $proxy_connection; 

and then i did:
service nginx restart

Now i can open onlyoffice-documents from nextcloud... but I'm still not sure if this bug is on traefik or onlyoffice side..

edited on 2019/11/29: service nginx restart .. but i think everyone did realize my mistake...

@senguendk
Copy link

i found a temporary solution by reconfiguring nginx inside the docker:

in file: /etc/nginx/includes/http-common.conf
i commented out this line (bottom of the file):

proxy_set_header Connection $proxy_connection; 
-->
# proxy_set_header Connection $proxy_connection; 

and then i did:
service restart nginx

Now i can open onlyoffice-documents from nextcloud... but I'm still not sure if this bug is on traefik or onlyoffice side..

I can confirm that this workaround works.

@Maurotb
Copy link

Maurotb commented Dec 5, 2019

I have the same issue since I upgraded Traefik to v2. /healthcheck/ returns true.

Same problem when update to traefik 2
Workarround # proxy_set_header Connection $proxy_connection; work for me

@LinneyS
Copy link
Member

LinneyS commented Dec 27, 2019

See the healthcheck status.
When setting up the proxy, you must transfer the protocol, host and port to the documentserver.
Look for errors while saving connection settings.
Enable full docuemtnserver and nextcloud logs and see error messages.

@arkanoid87
Could you please specify if the problem recurs?

@calvinbui
Copy link

For those on Traefik 2, I was able to fix the problem with these labels/headers:

traefik.http.routers.onlyoffice-secure.entrypoints: "web-secure"
traefik.http.routers.collabora-secure.tls: "true"
traefik.http.routers.collabora-secure.tls.certresolver: letsencrypt
traefik.http.services.onlyoffice.loadbalancer.server.port: "80"
traefik.http.routers.onlyoffice-secure.middlewares: "onlyoffice-sslheaders"
traefik.http.middlewares.onlyoffice-sslheaders.headers.customrequestheaders.X-Forwarded-Proto: "https"

image

Similarly if you're having CORS problems, use the CORS header as well

traefik.http.routers.onlyoffice-secure.middlewares: "onlyoffice-sslheaders, onlyoffice-cors"
traefik.http.middlewares.onlyoffice-sslheaders.headers.customrequestheaders.X-Forwarded-Proto: "https"
traefik.http.middlewares.onlyoffice-cors.headers.accessControlAllowOrigin: "*"

@wcdgit
Copy link

wcdgit commented Jan 16, 2020

Similarly if you're having CORS problems, use the CORS header as well

traefik.http.routers.onlyoffice-secure.middlewares: "onlyoffice-sslheaders, onlyoffice-cors"
traefik.http.middlewares.onlyoffice-sslheaders.headers.customrequestheaders.X-Forwarded-Proto: "https"
traefik.http.middlewares.onlyoffice-cors.headers.accessControlAllowOrigin: "*"

That's working for me, too. Thank you so much!

My docker-compose config:

version: '3'

services:
  onlyoffice:
    container_name: onlyoffice
    image: onlyoffice/documentserver:latest
    restart: unless-stopped
    stdin_open: true
    tty: true
    volumes:
      - ${LOCAL_CONF_DIR}/onlyoffice:/var/log/onlyoffice
    environment:
      JWT_ENABLED: "true"
      JWT_SECRET: ${SECRET}
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.${HOSTNAME}-http.entrypoints=web"
      - "traefik.http.routers.${HOSTNAME}-http.rule=Host(`${HOSTNAME}.${DOMAIN0}`)"
      - "traefik.http.routers.${HOSTNAME}-http.middlewares=https_redirect@file"
      - "traefik.http.routers.${HOSTNAME}-https.entrypoints=websecure"
      - "traefik.http.routers.${HOSTNAME}-https.rule=Host(`${HOSTNAME}.${DOMAIN0}`)"
      - "traefik.http.routers.${HOSTNAME}-https.tls=true"
      - "traefik.http.routers.${HOSTNAME}-https.middlewares=sts@file,onlyoffice-headers"
      - "traefik.http.services.${HOSTNAME}.loadbalancer.server.port=80"

      ## Middleware definition
      # Headers for onlyoffice, https://github.com/ONLYOFFICE/onlyoffice-nextcloud/issues/151
      - "traefik.http.middlewares.onlyoffice-headers.headers.customrequestheaders.X-Forwarded-Proto=https"
      - "traefik.http.middlewares.onlyoffice-headers.headers.accessControlAllowOrigin=*"

networks:
  default:
    external:
      name: ${NETWORK}

@derdrave
Copy link

does work for me too. thanks to @calvinbui and also to @wcdgit for the full docker-compose for checkup-purposes!

@leonfaeth
Copy link

@calvinbui 's labels fixed it for me, too. Thanks a lot! Can we somehow add a note or full example to https://helpcenter.onlyoffice.com/server/document/document-server-proxy.aspx to save people's time? The Traefik sample is still for version 1

@kungknut
Copy link

I can also confirm that adding the X-Forwarded-Proto header solved my issues.

@SuperSandro2000
Copy link

I can confirm that only the X-Forwarded-Proto Header is needed.

@0x9060
Copy link

0x9060 commented May 16, 2020

Similarly if you're having CORS problems, use the CORS header as well

traefik.http.routers.onlyoffice-secure.middlewares: "onlyoffice-sslheaders, onlyoffice-cors"
traefik.http.middlewares.onlyoffice-sslheaders.headers.customrequestheaders.X-Forwarded-Proto: "https"
traefik.http.middlewares.onlyoffice-cors.headers.accessControlAllowOrigin: "*"

That's working for me, too. Thank you so much!

My docker-compose config:

version: '3'

services:
  onlyoffice:
    container_name: onlyoffice
    image: onlyoffice/documentserver:latest
    restart: unless-stopped
    stdin_open: true
    tty: true
    volumes:
      - ${LOCAL_CONF_DIR}/onlyoffice:/var/log/onlyoffice
    environment:
      JWT_ENABLED: "true"
      JWT_SECRET: ${SECRET}
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.${HOSTNAME}-http.entrypoints=web"
      - "traefik.http.routers.${HOSTNAME}-http.rule=Host(`${HOSTNAME}.${DOMAIN0}`)"
      - "traefik.http.routers.${HOSTNAME}-http.middlewares=https_redirect@file"
      - "traefik.http.routers.${HOSTNAME}-https.entrypoints=websecure"
      - "traefik.http.routers.${HOSTNAME}-https.rule=Host(`${HOSTNAME}.${DOMAIN0}`)"
      - "traefik.http.routers.${HOSTNAME}-https.tls=true"
      - "traefik.http.routers.${HOSTNAME}-https.middlewares=sts@file,onlyoffice-headers"
      - "traefik.http.services.${HOSTNAME}.loadbalancer.server.port=80"

      ## Middleware definition
      # Headers for onlyoffice, https://github.com/ONLYOFFICE/onlyoffice-nextcloud/issues/151
      - "traefik.http.middlewares.onlyoffice-headers.headers.customrequestheaders.X-Forwarded-Proto=https"
      - "traefik.http.middlewares.onlyoffice-headers.headers.accessControlAllowOrigin=*"

networks:
  default:
    external:
      name: ${NETWORK}

This config is not working for me... I'm still getting (Connection refused) while connecting to upstream,

@jcgruenhage
Copy link

So, I've digged a bit, and the (merged) PR to traefik that @SuperSandro2000 linked above does not fix it. I refactored the code in traefik again to make it a bit clearer what's actually happening, but it doesn't make a difference really.

This needs to be fixed in ONLYOFFICE, but this is not the appropriate repository for that issue. Will open one though.

@zilexa
Copy link

zilexa commented Jan 10, 2021

I use FileRun (free, max 10 users, not opensource) instead of NextCloud, but I had the same issue with OnlyOffice.

For future reference, none of the workaround examples above are actually complete or are correct. After a few hours of trial and error, this is how I got it working.
I spend DAYS figuring this out as I am a Traefik and OO noob. And this Traefik forum topic send me searching in the dark wasting even more time: https://community.traefik.io/t/traefik-2-0-and-onlyoffice-not-work-work-correctly-in-v-2/3286

After I had https://office.mydomain working, I tested the example doc file. When that worked, I started removing several Traefik labels until I got the bare minimum set of extra rules necessary to run OnlyOffice successfully with Traefikv2.0.

##_____________________ OnlyOffice Document Server [Cloud/Office]
  onlyoffice:
    image: onlyoffice/documentserver
    container_name: onlyoffice
    depends_on:
      - onlyoffice-rabbitmq
    stdin_open: true
    restart: always
    tty: true
    ports:
      - "8889:80"
    volumes:
      - $USERDIR/docker/onlyoffice/data:/var/www/onlyoffice/Data
      - $USERDIR/docker/onlyoffice/log:/var/log/onlyoffice
      - $USERDIR/docker/onlyoffice/cache:/var/lib/onlyoffice/documentserver/App_Data/cache/files
      - $USERDIR/docker/onlyoffice/example:/var/www/onlyoffice/documentserver-example/public/files
      - $USERDIR/docker/onlyoffice/fonts:/usr/share/fonts
    dns: 1.1.1.1
    environment:
      - JWT_ENABLED="true"
      - JWT_SECRET=$ONLYOFFICEJWT
      - AMQP_URI=amqp://guest:guest@onlyoffice-rabbitmq
    labels:
     ## My standard traefikv2.0 labels for services exposed online:
      - traefik.enable=true
      - traefik.http.routers.office-redirect.entrypoints=web
      - traefik.http.routers.office-redirect.rule=Host(`office.$DOMAIN`)
      - traefik.http.middlewares.office-redirect.redirectscheme.scheme=https
      - traefik.http.routers.office.middlewares=office-redirect
      - traefik.http.routers.office-secure.entrypoints=websecure
      - traefik.http.routers.office-secure.rule=Host(`office.$DOMAIN`)
      - traefik.http.routers.office.tls.certresolver=letsencrypt
      - traefik.http.services.office.loadbalancer.server.port=80
      ## Extra labels for onlyoffice:
      - traefik.http.routers.office-secure.tls=true
      - traefik.http.routers.office-secure.middlewares=secure-headers
      - traefik.http.middlewares.secure-headers.headers.customrequestheaders.X-Forwarded-Proto=https
      - traefik.http.middlewares.cors-headers.headers.accessControlAllowOrigin=*
      - traefik.http.routers.office.middlewares=secure-headers,cors-headers


      ## tested extra labels, works fine without DO NOT USE
      #- traefik.http.middlewares.office-redirectregex.redirectregex.regex=^http://(.*)
      #- traefik.http.middlewares.office-redirectregex.redirectregex.replacement=https://$$1
      #- traefik.http.middlewares.secure-headers.headers.referrerPolicy=no-referrer
      #- traefik.http.middlewares.secure-headers.headers.stsSeconds=31536000
      #- traefik.http.middlewares.secure-headers.headers.forceSTSHeader=true
      #- traefik.http.middlewares.secure-headers.headers.stsPreload=true
      #- traefik.http.middlewares.secure-headers.headers.stsIncludeSubdomains=true
      #- traefik.http.middlewares.secure-headers.headers.browserXssFilter=true

Also:
I accidentally removed my postgreSQL container, discovered OnlyOffice still worked without it !?
Not sure how, but now OnlyOffice runs, just with rabbitmq as additional container. Perhaps the onlyoffice/documentserver image already contains a sql database:

##____________________ Onlyoffice rabbitmq [CLOUD/Office]
  onlyoffice-rabbitmq:
    container_name: onlyoffice-rabbitmq
    image: rabbitmq
    restart: always
    expose:
      - '5672'

@chmanie
Copy link

chmanie commented Apr 25, 2021

@wcdgit and @calvinbui thanks so much for sharing your configs! Sadly these are still not working for me. This is my current config:

  onlyoffice-document-server:
    container_name: nextcloud-onlyoffice
    image: onlyoffice/documentserver:latest
    restart: always
    expose:
      - '80'
      - '443'
    volumes:
      - document_data:/var/www/onlyoffice/Data
      - document_log:/var/log/onlyoffice
    networks:
      - traefik_default
    labels:
      - traefik.enable=true
      - traefik.http.routers.onlyoffice-document-server.rule=Host(`-SNIP-`)
      - traefik.http.routers.onlyoffice-document-server.entrypoints=web
      - traefik.http.routers.onlyoffice-document-server.middlewares=https-redirect
      - traefik.http.routers.onlyoffice-document-server-https.rule=Host(`-SNIP-`)
      - traefik.http.routers.onlyoffice-document-server-https.entrypoints=websecure
      - traefik.http.routers.onlyoffice-document-server-https.tls=true
      - traefik.http.routers.onlyoffice-document-server-https.tls.certresolver=letsencrypt
      - traefik.http.routers.onlyoffice-document-server-https.middlewares=onlyoffice-headers
      - traefik.http.middlewares.onlyoffice-headers.headers.customrequestheaders.X-Forwarded-Proto=https
      - traefik.http.middlewares.onlyoffice-headers.headers.accessControlAllowOrigin=*

This is the config of the nextcloud app:

image

Here's the config for the nginx between traefik and nextcloud (and onlyoffice): https://gist.github.com/chmanie/3411b3533bbcfd3dd55a33a18accd31f

Am I missing anything?

EDIT: Commenting out the link as mentioned here worked for me, but I don't like this as a permanent solution.

@zilexa
Copy link

zilexa commented Apr 25, 2021

Just fyi since I posted my Traefik solution I switched to Caddyv2 as it is extremely simplified compared to Traefik. This works fine:

https://github.com/zilexa/Homeserver/blob/master/docker/docker-compose.yml#L279
With the caddy-docker-proxy container (also in that compose example). No other configuration is needed.

I also have a fully tested example with Nextcloud instead of FileRun:
https://github.com/zilexa/Homeserver/blob/master/docker/Extras/nextcloud.yml

Both cases A+ security rating instantly.

@MTRNord
Copy link

MTRNord commented Jan 19, 2022

Any update on this? Neither the custom request header nor removing the line in the nginx config works for me using the onlyoffice helm setup and a traefik2 ingress. :/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests