Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restoring backups via XO proxy does not work if it takes longer than 5 minutes #7052

Closed
darkowls opened this issue Sep 22, 2023 · 0 comments · Fixed by #7152
Closed

Restoring backups via XO proxy does not work if it takes longer than 5 minutes #7052

darkowls opened this issue Sep 22, 2023 · 0 comments · Fixed by #7152

Comments

@darkowls
Copy link

Are you using XOA or XO from the sources?

XOA Enterprise and XOA Proxy

If XOA:

  • which release channel? (stable vs latest)
    stable
  • please consider creating a support ticket in your dedicated support area
    Done that 01.09.2023
    Describe the bug
    When using xo-proxy for create backups - everything works fine.
    When using xo-proxy to restore backups - every time it stop exactly after 5 minutes
    Restore works when using XOA without proxy or when VM is smaller and get done in less than 5 minutes

xo logs on proxy state:
Sep 22 14:52:51 bak-xoproxy-ng04.sl-int.team xo-proxy[11262]: root@65.21.133.79 Xapi#putResource /import/ Error: Premature close
Sep 22 14:52:51 bak-xoproxy-ng04.sl-int.team xo-proxy[11262]: at new NodeError (node:internal/errors:393:5)
Sep 22 14:52:51 bak-xoproxy-ng04.sl-int.team xo-proxy[11262]: at ClientRequest.onclose (node:internal/streams/end-of-stream:147:30)
Sep 22 14:52:51 bak-xoproxy-ng04.sl-int.team xo-proxy[11262]: at ClientRequest.emit (node:events:525:35)
Sep 22 14:52:51 bak-xoproxy-ng04.sl-int.team xo-proxy[11262]: at ClientRequest.patchedEmit [as emit] (/usr/local/lib/node_modules/@xen-orchestra/proxy/node_modules/@xen-orchestra/log/configure.js:52:17)
Sep 22 14:52:51 bak-xoproxy-ng04.sl-int.team xo-proxy[11262]: at TLSSocket.socketCloseListener (node:_http_client:457:9)
Sep 22 14:52:51 bak-xoproxy-ng04.sl-int.team xo-proxy[11262]: at TLSSocket.emit (node:events:525:35)
Sep 22 14:52:51 bak-xoproxy-ng04.sl-int.team xo-proxy[11262]: at TLSSocket.patchedEmit [as emit] (/usr/local/lib/node_modules/@xen-orchestra/proxy/node_modules/@xen-orchestra/log/configure.js:52:17)
Sep 22 14:52:51 bak-xoproxy-ng04.sl-int.team xo-proxy[11262]: at node:net:313:12
Sep 22 14:52:51 bak-xoproxy-ng04.sl-int.team xo-proxy[11262]: at TCP.done (node:_tls_wrap:587:7)
Sep 22 14:52:51 bak-xoproxy-ng04.sl-int.team xo-proxy[11262]: at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {
Sep 22 14:52:51 bak-xoproxy-ng04.sl-int.team xo-proxy[11262]: code: 'ERR_STREAM_PREMATURE_CLOSE'
Sep 22 14:52:51 bak-xoproxy-ng04.sl-int.team xo-proxy[11262]: }

To Reproduce

  1. Create backup larger than 100 Gb on 1 gbit network.
  2. Try to restore it using xo-proxy
  3. Look how VHD are created on xcp-ng and get larger as import goes.
  4. Wait with the stopwatch for exactly 5 minutes.
  5. Watch the VHD get deleted on xcp-ng, and task fails

Expected behavior
Backups of larger VMs being restored even if it takes more than 5 minutes
Actually - it did work right after the deployment once, but after some time it stopped working.

Screenshots

изображение

Environment (please provide the following information):

XOA version:
xo-server 5.120.2
xo-web 5.122.2

Proxy:
[15:00 22] bak-xoproxy-ng04:$ xoa check
✔ Node version
✔ Memory
✔ xo-server config syntax
✔ Disk space for /
✔ Disk space for /var
✔ Native SMB support
✔ Fetching VM UUID
✔ XOA version
✔ /var is writable
✔ Appliance registration
✔ xoa-support user
✔ NTP synchronization
✔ local SSH server
✔ Internet connectivity
✔ npm version
✔ XOA status
[15:00 22] bak-xoproxy-ng04:
$ node -v
v18.12.1

  • hypervisor: XCP-ng 8.2.1 fully patched

Additional context
I tried to apply http timeout patch (it did not help, as it does not work on node JS 18)
Tried updating XOA and proxy via Web Interface.
Tried redeploying proxy
Tried updating node JS to latest 18 and 20
Tried to add options to nano /etc/xo-server/config.toml
[http.listenOptions]
requestTimeout = 0

Tried reading trough all the forums on XOA and Node JS (like # nodejs/node#46574)

Tried building XO and proxy from the latest sources

@julien-f julien-f self-assigned this Sep 25, 2023
@olivierlambert olivierlambert added this to the Q4 2023 milestone Oct 13, 2023
julien-f added a commit that referenced this issue Nov 2, 2023
Fixes #7052

Fixes zammad#17383

When a stream is returned, the handler immediately returned a stream which disposed the resource.

Due to the disposable having a 5 mins debounce delay, the problem was only apparent after 5 mins.
julien-f added a commit that referenced this issue Nov 6, 2023
Fixes #7052

Fixes zammad#17383

When a stream is returned, the handler immediately returned a stream which disposed the resource.

Due to the disposable having a 5 mins debounce delay, the problem was only apparent after 5 mins.
julien-f added a commit that referenced this issue Nov 6, 2023
…7152)

Fixes #7052

Fixes zammad#17383

When a stream is returned, the handler immediately returned a stream which disposed the resource.

Due to the disposable having a 5 mins debounce delay, the problem was only apparent after 5 mins.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment