File restore not possible from older version

[john-arvid]

Context

• XO origin: the sources
• Versions (from were the backups were taken):
  • Node: 8.12.0
  • xo-web: xo-web 5.35.0
  • xo-server: xo-server 5.35.0
• Versions (from were the backups were tried to be restored):
  • Node: 8.12.0
  • xo-web: xo-web 5.46.0
  • xo-server: xo-server 5.46.0

Expected behavior

Expect file restore to be possible from backups done with old versions of Orchestra.

Current behavior

I can chose backup date, disk and partition, but the files are not showing up.

Other information

I am using the Backup NG with Delta backups.
I don't get any errors in the web interface.

Let me know what information I should gather for this issue.

[olivierlambert]

Hi,

Can you check with XOA to see if it's an environment problem or a XO bug? You can use this to deploy it easily: https://xen-orchestra.com/#!/xoa

[john-arvid]

Hi,

I have installed XOA and it worked as expected there.
Do you think I should try to install the same version from the sources as XOA is running?

[olivierlambert]

No, I think it's an environment error, something related on the way it's running, not due to a XO version.

Have you followed our doc for the source install? https://xen-orchestra.com/docs/from_the_sources.html

[john-arvid]

Yes when I installed it maybe 6 months ago. I have just used the update section in the docs after that.
I can start a new vm if you want me to test that.

[olivierlambert]

Try with a fresh install following instructions, and ideally on a Debian VM to be close as possible of XOA 👍

[john-arvid]

I have now installed a new vm, debian 9 with all updates. Then installed from the sources with the instructions from the docs.
The problems is the same.

I see now that I get a message in the vm's console after I chose partition
ntfs: (device loop0): parse_options(): Unrecognized mount option norecovery.

[john-arvid]

I reverted back to version 5.44, the same as XOA is running and now it is working. So it looks like the problem is with the newer versions.

[olivierlambert]

Can you point since which commit it seems to work?

[john-arvid]

I will try, do you have a simple way I can upgrade one commit at the time?

[olivierlambert]

In theory you have git bisect (see https://git-scm.com/docs/git-bisect) but you can do that manually: go from the latest working commit you know that works, then find a commit halfway to the latest one, and select it (git checkout <commit ID>).

Do your tests, and if it works: this commit is now the new "base", so go from there up to halfway of the latest commit. And repeat.

If you "land" on a commit where it starts to fail, go backward.

Finally, you'll find the culprit 😄

[john-arvid]

I have gone through some of the commits in the releases.
This is the last good one: ff9782b
This one causes problems: 82fec86

[olivierlambert]

Can you go back on your XOA and select latest channel, then update, and tell me if you have the issue or not?

[john-arvid]

Done that, it's not a problem on XOA.

[olivierlambert]

Even on latest channel? Because latest XOA is running on something more recent than the commit in question. To see on which channel you are, go on "XOA" menu entry, select latest, then update.

[john-arvid]

That is correct, I changed from stable to latest channel. Current version: 5.37.0

[olivierlambert]

Okay so in XOA you have something more recent than the commit you tested with a problem, I don't know why.
Can you try to checkout on 516edd1b09b638f70edcbd497e6715bf51627d44 and tell me if it works on your source install?

[john-arvid]

Tested, same problem as before.

[olivierlambert]

I have no idea then. @julien-f will take a look, but if it works on a more recent commit in XOA, it's a environment issue.

[john-arvid]

Yeah I agree to that. But there is something done after ff9782b that made the environment give back the error: ntfs: (device loop0): parse_options(): Unrecognized mount option norecovery.

[olivierlambert]

So to recap: right after this commit, it doesn't work for you?

[olivierlambert]

You are on master branch right?

[john-arvid]

Yes, the one after that commit in the releases list causes that problem.
I would guess so, I am using the commands from the docs.

[olivierlambert]

To be sure, can you tell me the exact commit causing the issue? For it, it seems a very small commit without anything related to your issue.

[john-arvid]

I have only tested these. Want me to test more releases in the release list?

I have gone through some of the commits in the releases.
This is the last good one: ff9782b
This one causes problems: 82fec86

[olivierlambert]

The problematic commit is just jumping a version of xen-api, which is not linked to your issue. But I'll let @julien-f taking a look when he's back from holidays (around mid-august)

[olivierlambert]

In the meantime, feel free to retry when we have new commits on master

[john-arvid]

No, there was more than just that happening between the releases. As I said, I was using the releases list.
But now I have gone thorugh the commits, and this is the commit that causes a problem: efdfa1f
And this is the last one the is working: 5bd61e3

When I say releases list, I mean this: https://github.com/vatesfr/xen-orchestra/releases
When I say commit list, I mean this: https://github.com/vatesfr/xen-orchestra/commits/master

And it looks like the error: ntfs: (device loop0): parse_options(): Unrecognized mount option norecovery. doesn't have much to do with this. I saw it in the last working commit also.

I will try new commits now and then to see if it start to work again while I wait for @julien-f

[olivierlambert]

Okay so from the sources, I don't think it's a good idea to rely on GitHub releases: because it's complicated and not meant to be used like that. In XOA, we do a specific integration of what's better between different components (running a lot of QA of various component version to craft a stable XOA). In short, XOA is really like a distro dedicated to Xen Orchestra.

If you don't want to do that work yourself, I would strongly suggest to just rely on master branch (or use XOA). Also, it's far easier to make community assistance if you are on master than on any other configuration possible (almost infinite combinations between each components).

Regarding the commit details, it's out of my area of expertise, so @julien-f will the man to ask!

[john-arvid]

Ok, but it looks like all the releases in GitHub release is from the masters branch.
But either way, this was just done now to find where the problem started. Normally I think people pulls the latest commit from the master branch and thats it.

I will wait for @julien-f

[john-arvid]

In my case I have done that, the only thing that is the same between rebuilds, is the SR with the backups.

[olivierlambert]

And those very same backups can be restored from an XOA, right? Trying to figure out this, it doesn't make sense 😕

[mmartinWECHU]

And those very same backups can be restored from an XOA, right? Trying to figure out this, it doesn't make sense 😕

It doesn't look like it mounts the image in the same way. I don't believe the backup NG mounts the image for an SR restore. I could be wrong.

After a rebuild with most current XOA from source I still have the same issue with file restore.

Is there any logs or items that I can provide that would be helpful?

[olivierlambert]

There's no XO code difference with XOA, it's the entire environment that's tested, that's the main diff (not the code).

That's why I'm puzzled with your issue. Check your browser console log while you try to restore.

[mmartinWECHU]

I agree it must be something in the environment. I was wondering if you would like some log collection to see what it is. Since environment is outside of XO from source scope I understand if I need to deal with this on my own.

[john-arvid]

@olivierlambert Yes, with XOA it works ok. I can try to check the browser logs a bit later.

But regarding the environment, I tested as @olivierlambert said, Debian 9, all updates, followed the instructions in the docs. So it should be the relatively the same as in a XOA environment.

[mmartinWECHU]

From the browser:

Unhandled promise rejection TypeError: "invalid assignment to const `n'"
    _listFiles restore-file-modal.js:44
    i es6.promise.js:75
    R es6.promise.js:92
    u _microtask.js:18
es6.promise.js:110
exports _perform.js:3
    z es6.promise.js:104
    exports _invoke.js:5
    <anonymous> _task.js:35
    g _task.js:21
    y _task.js:25

​
​

[olivierlambert]

What's the node and npm version do you have?

[Danp2]

I'm getting the same browser error as @mmartinWECHU. Previously posted about it here.

[olivierlambert]

@john-arvid thing is, there's a difference somewhere because we can't reproduce it, neither in XOA nor in our dev env.

@julien-f any idea on this after seeing those browser logs?

[Danp2]

@olivierlambert Still seeing this after upgrading to latest available source. Here's my details --

root@ubuntutest:~# node -v
v8.16.0
root@ubuntutest:~# npm -v
6.4.1
root@ubuntutest:~# yarn -v
1.17.3

root@ubuntutest:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 19.04
Release:        19.04
Codename:       disco

[julien-f]

@julien-f any idea on this after seeing those browser logs?

This is not uninteresting but not really helpful because the code is minified so lines and variable names don't mean anything.

[Danp2]

@julien-f I'm willing to help isolate this issue. Let me know if there's anything I can do on this end.

[Rockz1152]

Retested with latest commit: ee9cc05

Still seeing this issue

root@xo:~# node -v
v8.16.1
root@xo:~# npm -v
6.4.1
root@xo:~# yarn -v
1.17.3

root@xo:~# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 9.9 (stretch)
Release:        9.9
Codename:       stretch

Browser console output when trying to restore a file from delta backup

Unhandled promise rejection TypeError: "invalid assignment to const `n'"
    _listFiles restore-file-modal.js:44
    i es6.promise.js:75
    R es6.promise.js:92
    u _microtask.js:18
es6.promise.js:110
    t es6.promise.js:110
    exports _perform.js:3
    z es6.promise.js:104
    exports _invoke.js:5
    <anonymous> _task.js:35
    g _task.js:21
    y _task.js:25

[flipsa]

I am also affected by this. And as many here have said already, I agree that efdfa1f is the commit that breaks things.

I tried reverting this commit and then rebuilt XO, and I can now restore files again. But it's obviously an ugly hack with potential side-effects (esp. for people who still use nodejs-v6 - if I interpret the commit message correctly).

In effect, I now run the latest XO, but with somewhat older npm dependencies, and so far I didn't experience any problems, but it's only been a few hours and just some basic tests...

Did anybody else try this and found anything that breaks?

[Danp2]

This is what I'm getting in the Chrome console --

restore-file-modal.js:44 Uncaught (in promise) TypeError: Assignment to constant variable.
    at then.setState.scanningFiles (restore-file-modal.js:44)
then.setState.scanningFiles @ restore-file-modal.js:44
Promise.then (async)
_listFiles @ restore-file-modal.js:93
e.notifyAll @ CallbackQueue.js:74
close @ ReactUpdates.js:57
closeAll @ Transaction.js:207
perform @ Transaction.js:154
perform @ ReactUpdates.js:87
S @ ReactUpdates.js:170
closeAll @ Transaction.js:207
perform @ Transaction.js:154
batchedUpdates @ ReactDefaultBatchingStrategy.js:60
batchedUpdates @ ReactUpdates.js:95
dispatchEvent @ ReactEventListener.js:145

[flipsa]

@Danp2 this looks like the same error than before. It's still tripping over constructing the correct file path...

While investigating further, I did this a few times by now, and so far it worked every time, also on a freshly installed system. The steps needed:

• git revert efdfa1f (or just replace your yarn.lock with this one)
• yarn && yarn build
• restart XO

However, since you are on a different distro / version (albeit very similar), YMMV...

[Danp2]

@flipsa Yes, I've done this as well. It seems odd that the devs are not able to reproduce this. I was hoping that providing the addition details might assist @julien-f with diagnosing the issue.

[Danp2]

Ok... so here's a temporary fix until the devs can figure this out. Edit the file ./packages/xo-web/src/xo-app/backup-ng/file-restore/restore-file-modal.js and change the line

const { backup, disk, partition, path } = this.state

to

let { backup, disk, partition, path } = this.state

After that, execute the command yarn build and then restart xo-server.

[Danp2]

For anyone interested, this is a section of code from the index.js of xo-web post "yarn build" --

this._listFiles=(()=>{const{backup:e,disk:t,partition:r,path:n}=this.state;return this.setState({scanningFiles:!0}),(0,w.listFiles)(e.remote.id,t,n,r).then(e=>this.setState({files:(e=e,n=n,("/"!==n?[{id:"..",isFile:!1,name:"..",path:(e=>e+(e.endsWith("/")?"":"/"))((0,b.dirname)(n))}]:[]).concat((0,o.default)(e,(e,t)=>({id:`${n}${t}`,isFile:!t.endsWith("/"),name:t,path:`${n}${t}`})))),scanningFiles:!1,listFilesError:!1}),e=>{throw this.setState({scanningFiles:!1,listFilesError:!0}),e})}),this._getSelectableFiles=(0,y.createSelector)(()=>this.state.files,()=>this.state.selectedFiles,(e,t)=>(0,u.default)(e,e=>!(0,i.default)(t,e))),this._onBackupChange=(e=>{this.setState({backup:e,disk:void 0,partition:void 0,file:void 

Note that e and n is declared as a constants representing backup and path. Later on, both e and n are assigned to themselves, which appears to be the problematic section of code from formatFilesOptions.

[julien-f]

Thank you all for your help investigating this! ❤️

[john-arvid]

Great work!
But why wasn't this a problem with XOA?

[julien-f]

Because the packaging system is quite different and was not using the exact same version of the dependency. 😭