[stable9] Backport lazy init of shared storage/mount #26912

PVince81 · 2017-01-10T09:52:39Z

Description

Backport of 2f1c62c from SharedStorage to new sharing code + cleanup #23919 to introduce LazyStorageMountInfo
Backport of c8a385d from [stable9.1] Fix sharedstorage recursion hell #25789 to have a shortcut to retrieve the numeric storage id without having to initialize the shared storage fully
Backport of dc7d55c to have an additional safety against unwanted recursions in shared storages (could not make it happen on this PR but the extra safety won't harm)

Related Issue

Hopefully fixes the issue described in #26702

Motivation and Context

There's a perf regression in 9.0 that didn't exist in 8.2 due to the introduction of oc_mounts.
Perf was improved in 9.1 through #25789 so hopefully backporting a subset of it will also improve performance in 9.0.

How Has This Been Tested?

Steps:

Setup OC 9.0
Run https://gist.github.com/PVince81/4c4c2e938ae29065dcf02cdc2cc6d0bc (requires latest pyocclient)
curl -D - -X GET -u recipient:recipient http://localhost/owncloud/remote.php/webdav/file1.txt > file.txt => ignore the result
curl -D - -X GET -u recipient:recipient http://localhost/owncloud/remote.php/webdav/file1.txt > file.txt, again, and have a look at the value of "Time spent"

Before this fix (v9.0.7): time spent was 7 seconds
After this fix: time spent is down to 4 seconds

Running the same test on v9.1.3 also gives 4 seconds.

Screenshots (if appropriate):

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.
I have read the CONTRIBUTING document.
I have added tests to cover my changes.
All new and existing tests passed.

TODOs:

⚠️ it is likely that the introduction of this laziness will cause a regression due to a new recursive behavior (this was observed on 9.1, need to confirm if it will appear as well here through this PR). This was fixed by the PR #25789 which itself brought its own regressions fixed by subsequent PRs.

check if there is a simpler shortcut => yes, but might not be worth it if we need additional backports anyway due to laziness (shortcut is on branch "stable9-shortcut-sharedmount-ids")
analyze whether this PR now contains the shared storage recursion hell and requires further backports => no recursion observed, but still did an extra safety backport
no further backports needed as this is not a Jail instance, so no further regression that were observed on 9.1 would happen
@mrow4a please rerun your powerful tests on this branch

mention-bot · 2017-01-10T09:52:40Z

@PVince81, thanks for your PR! By analyzing the history of the files in this pull request, we identified @butonic and @DeepDiver1975 to be potential reviewers.

PVince81 · 2017-01-10T10:12:36Z

Good news: the SharedStorage in stable9 is not a Jail, so no need to backport the jail fix from 3717e62.

Now there is still a risk that the recursion happens when calling $this->ownerView->getPath(), I'll debug into it to see if it happens. If yes, we'll need dc7d55c

also backport the FailedStorage case, for more robustness ? 837bf1c => no

PVince81 · 2017-01-10T10:41:49Z

Attempting to find steps to reproduce the recursion in question, because we never really had steps for that... Here #25557 (comment)

PVince81 · 2017-01-10T12:08:26Z

Could not reproduce the infinite recursion on this PR here #25557 (comment).

It is likely that the code is too different. But there is still a slight chance that some code paths triggers it, so I decided to backport the recursion safety commit as well as it won't hurt other code paths: dc7d55c

The FailedStorage backport looks unnecessary as the SharedStorage is not a Jail instance like on 9.1 where a lot of logic is different.

I think this PR is now ready for review and testing.

PVince81 · 2017-01-10T12:15:15Z

@mrow4a this is ready for testing, please run your magic tests on this branch. Thanks

PVince81 · 2017-01-11T12:50:09Z

Looks like e6d8779 breaks some unit tests:

	Test Result (2 failures )

    Test_Files_Sharing_Api.testGetShareFromFolderReReShares
    Test_Files_Sharing_Api.testGetShareFromFileReReShares

PVince81 · 2017-01-11T13:03:25Z

This because the tests create shares the old way which doesn't create flat reshares.
Tried to backport 6509220 to have it use the proper APIs, but it makes the tests fail differently now.

PVince81 · 2017-01-11T17:11:03Z

I had to compile a few backports to make tests pass again. The reason is that the getPath() shortcut is based on the assumption that reshares are flat. But the unit tests use the old API which uses the old chain-like shares. Note that the old API isn't used any more.

REQUIRES: this other backport PR [stable9] Backport flat reshare unit tests #26927 to be merged first before tests can pass here.

PVince81 · 2017-01-11T18:13:25Z

Looks like I'll have to figure out how to run the complex performance tests myself...

Can I at least get a code review ?

PVince81 · 2017-01-11T18:39:59Z

Also, am running smashbox against this branch, just in case.

mrow4a · 2017-01-11T18:40:06Z

@PVince81 I have everything set up, OC server with different versions, smashbox appliance etc. I just need to checkout the branch on one of server VM, put correct config files and run the tests on "smashbox" VM.

Everything is there, I will try to finish quickly university stuff and help you -> https://malibu.int.owncloud.com:8006/ on internal network, password you know,

PVince81 · 2017-01-11T18:40:51Z

@mrow4a thanks a lot!

phisch · 2017-01-12T12:05:14Z

lib/private/files/config/usermountcache.php

-			if ($storage->instanceOfStorage('\OC\Files\Storage\Shared')) {
-				$rootId = (int)$storage->getShare()['file_source'];
+			// note: SharedMount goes there
+			if (method_exists($mount, 'getStorageRootId')) {


Why is this necessary? IMountPoint has this method, if it does not exist you would either get an error for something not implementing IMountPoint correctly, or not injecting an object of type IMountPoint.

IMountPoint only has this method on stable9.1 or master and we don't want to introduce new methods as it qualifies as API change. Other implementers of IMountPoint (ex: external storage) will likely not implement this one in this version. So this hack is there to accomodate for that.

Ah, i didn't think about that! Makes sense then!

To prevent recursions in initMountPoints which requires the numeric id to populate oc_mounts

PVince81 · 2017-01-12T14:55:52Z

Rebased to include #26927, tests should pass now.

PVince81 · 2017-01-13T09:59:10Z

@mrow4a has just confirmed that this improves performance significantly, and some results aren't exponential any more. The results aren't too close to master, but good enough to be released.

Please review the code @PhilippSchaffrath @jvillafanez

This PR will also need some regression testing @owncloud/qa

jvillafanez · 2017-01-16T07:55:47Z

apps/files_sharing/lib/sharedmount.php

@@ -58,9 +61,10 @@ class SharedMount extends MountPoint implements MoveableMount {
 	public function __construct($storage, array $mountpoints, $arguments = null, $loader = null) {
 		$this->user = $arguments['user'];
 		$this->recipientView = new View('/' . $this->user . '/files');
-		$newMountPoint = $this->verifyMountPoint($arguments['share'], $mountpoints);
+		$this->share = $arguments['share'];


We should check if the "share" key is present and throw an error otherwise.

jvillafanez · 2017-01-16T08:02:16Z

lib/private/files/config/lazystoragemountinfo.php

+	 */
+	public function __construct(IUser $user, IMountPoint $mount) {
+		$this->user = $user;
+		$this->mount = $mount;


I think we have to call the parent constructor. The parent might not be properly initialized otherwise and could cause issues.
If this is intended, write a comment explaining it.

This was backported from 9.1 which didn't call the parent either (not my code), I hope it had a good reason

jvillafanez · 2017-01-16T08:28:33Z

lib/private/files/view.php

@@ -1663,6 +1665,11 @@ public function getPath($id) {
 			/**
 			 * @var \OC\Files\Mount\MountPoint $mount
 			 */
+			if (!$includeShares && $mount instanceof SharedMount) {


I'm not sure if this will be enough to prevent the infinite loop

This was enough on 9.1, at least for known cases

jvillafanez · 2017-01-16T08:48:16Z

I think we'll have to review the lazystoragemountinfo.php the sooner the better. It looks like a bad design, and this kind of things are always difficult to get rid of if they're left alone for a long time.

First I think it's pointless to have both classes (CachedMountInfo and LazyStorageMountInfo): you'll do it lazily or not, but you won't do it lazily sometimes and then normal.

Second, usually the class on top (CachedMountInfo in this case) is the one that has control over what the subclasses are allowed to do, and only a few methods are overwritten in the subclasses. Taking into account that we have control over both classes (CachedMountInfo and LazyStorageMountInfo), there is no reason not to modify the CachedMountInfo accordingly to allow lazyness in a clearer way, instead of overwrite almost all the methods and using protected parameters that aren't documented as "allowed to be used this way" in the subclasses or something similar.

I don't expect to solve this in this PR because it's a backport, but we have to take this into account.

PVince81 · 2017-01-16T16:45:50Z

First I think it's pointless to have both classes (CachedMountInfo and LazyStorageMountInfo): you'll do it lazily or not, but you won't do it lazily sometimes and then normal.

You're right. I'm also not too happy about these. Would you suggest merging both classes into a single one ? (in a separate tech debt PR) That would likely solve all points you mentionned (calling parent, always lazy and no weird control)

jvillafanez · 2017-01-17T08:28:44Z

Users won't care about the specific implementation as long as it's fast enough, and a don't think there are cases where one implementation would perform better than the other one in order to provide a switch for those cases, so one implementation is enough.

PVince81 · 2017-01-18T10:33:57Z

@jvillafanez I added the required checks. Your LazyCacheMountInfo can be adressed separately on master as tech debt: #26960

jvillafanez · 2017-01-18T16:40:27Z

👍

lock · 2019-08-04T02:02:49Z

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

PVince81 added the 2 - Developing label Jan 10, 2017

PVince81 added this to the 9.0.8 milestone Jan 10, 2017

PVince81 mentioned this pull request Jan 10, 2017

MountManager::find called too often for the same entries #26702

Closed

PVince81 force-pushed the stable9-sharedstoragehell-backport branch from b0326cc to e6d8779 Compare January 10, 2017 12:09

PVince81 added 3 - To Review and removed 2 - Developing labels Jan 10, 2017

PVince81 requested review from jvillafanez, mrow4a and butonic January 10, 2017 12:15

PVince81 added the green-ticket label Jan 10, 2017

This was referenced Jan 11, 2017

[stable9] Backport flat reshare unit tests #26926

Closed

[stable9] Backport flat reshare unit tests #26927

Merged

phisch reviewed Jan 12, 2017

View reviewed changes

icewind1991 and others added 3 commits January 12, 2017 15:53

Only construct the storage when we start using it

2b6b2dc

Get shared storage numeric id directly from DB

a55da70

To prevent recursions in initMountPoints which requires the numeric id to populate oc_mounts

Flag to not recurse into shared mounts in getPath

32bde45

PVince81 force-pushed the stable9-sharedstoragehell-backport branch from e6d8779 to 32bde45 Compare January 12, 2017 14:54

jvillafanez reviewed Jan 16, 2017

View reviewed changes

Check arguments in SharedMount constructor

b26aa0f

PVince81 mentioned this pull request Jan 18, 2017

Merge LazyCacheMountInfo with CachedMountInfo #26960

Closed

PVince81 merged commit 72f0fd6 into stable9 Jan 18, 2017

PVince81 deleted the stable9-sharedstoragehell-backport branch January 18, 2017 16:44

lock bot locked as resolved and limited conversation to collaborators Aug 4, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[stable9] Backport lazy init of shared storage/mount #26912

[stable9] Backport lazy init of shared storage/mount #26912

PVince81 commented Jan 10, 2017 •

edited

Loading

mention-bot commented Jan 10, 2017

PVince81 commented Jan 10, 2017 •

edited

Loading

PVince81 commented Jan 10, 2017

PVince81 commented Jan 10, 2017

PVince81 commented Jan 10, 2017

PVince81 commented Jan 11, 2017

PVince81 commented Jan 11, 2017

PVince81 commented Jan 11, 2017 •

edited

Loading

PVince81 commented Jan 11, 2017

PVince81 commented Jan 11, 2017

mrow4a commented Jan 11, 2017 •

edited

Loading

PVince81 commented Jan 11, 2017

phisch Jan 12, 2017

PVince81 Jan 12, 2017

phisch Jan 12, 2017

PVince81 commented Jan 12, 2017

PVince81 commented Jan 13, 2017

jvillafanez Jan 16, 2017

PVince81 Jan 18, 2017

jvillafanez Jan 16, 2017

PVince81 Jan 16, 2017

jvillafanez Jan 16, 2017

PVince81 Jan 16, 2017

jvillafanez commented Jan 16, 2017

PVince81 commented Jan 16, 2017

jvillafanez commented Jan 17, 2017

PVince81 commented Jan 18, 2017

jvillafanez commented Jan 18, 2017

lock bot commented Aug 4, 2019

[stable9] Backport lazy init of shared storage/mount #26912

[stable9] Backport lazy init of shared storage/mount #26912

Conversation

PVince81 commented Jan 10, 2017 • edited Loading

Description

Related Issue

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

Checklist:

TODOs:

mention-bot commented Jan 10, 2017

PVince81 commented Jan 10, 2017 • edited Loading

PVince81 commented Jan 10, 2017

PVince81 commented Jan 10, 2017

PVince81 commented Jan 10, 2017

PVince81 commented Jan 11, 2017

PVince81 commented Jan 11, 2017

PVince81 commented Jan 11, 2017 • edited Loading

PVince81 commented Jan 11, 2017

PVince81 commented Jan 11, 2017

mrow4a commented Jan 11, 2017 • edited Loading

PVince81 commented Jan 11, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PVince81 commented Jan 12, 2017

PVince81 commented Jan 13, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jvillafanez commented Jan 16, 2017

PVince81 commented Jan 16, 2017

jvillafanez commented Jan 17, 2017

PVince81 commented Jan 18, 2017

jvillafanez commented Jan 18, 2017

lock bot commented Aug 4, 2019

PVince81 commented Jan 10, 2017 •

edited

Loading

PVince81 commented Jan 10, 2017 •

edited

Loading

PVince81 commented Jan 11, 2017 •

edited

Loading

mrow4a commented Jan 11, 2017 •

edited

Loading