Handle owncloud migration to latest release #23044

juliusknorr · 2020-09-25T10:11:26Z

Steps to migrate from 10.5 to this branch

make sure to have activity/files_pdfviewer/notifications/viewer/firstrunwizard installed
occ upgrade
occ db:convert-filecache-bigint
occ db:add-missing-columns
occ db:add-missing-indices
occ db:add-missing-primary-keys (edit by PVince81, had to do this on my setup)

Adjustments

Avoid exception in bruteforce protection capabilities on the upgrade page (as the table doesn't exist yet)
db:add-missing-indices
In SchemaException.php line 85: There is no column with name 'userid' on table 'oc_properties'.
Check for removed migrations/repair in server steps between 12 and master
Revert repair steps that might be relevant when migrating

Missing from the database diff between 10.5 and 20

With this set of patches the migration succeed however there is some difference in the database schema that should be addressed:

Removing data from database (core/Migrations/Version21000Date20201120141228.php)

oc_dav_job_status -> delete this table, see Handle owncloud migration to latest release #23044 (comment)
oc_systemtags -> remove assignable
oc_share -> remove attributes

Pulled out

oc_federated_reshares -> move remote_id to string (Use string for storing a OCM remote id #24247)
oc_federated_reshares -> share_id to bigint (Use string for storing a OCM remote id #24247)
oc_activity -> app nc: varchar(32) oc: varchar(255) , add type to activity_filter key
oc_notifications -> only index names are different

PVince81 · 2020-10-29T13:22:04Z

oc_activity -> app varchar 32 -> 255, add type to activity_filter key

In a fresh install of NC 20 the app column still has 32 characters, same in activity master.
Not sure where that comes from.

PVince81 · 2020-10-29T13:51:00Z

ok, got it, it's the other way around...

PVince81 · 2020-10-29T14:46:56Z

regarding bigints I think we should add these to NC (and not convert bigint to int in NC)
the reason is most likely that the number of fileids/entries can become very high on huge instances

not sure if such change would be backported to NC 20 (if DB changes are allowed in minor releases), so might only arrive with NC 21 then ?

PVince81 · 2020-10-29T14:50:25Z

oc_dav_job_status is for pending asynchronous/lazy operations: https://github.com/owncloud/core/blob/master/apps/dav/lib/DAV/LazyOpsPlugin.php#L65

we can delete it as it shouldn't contain any active entries during a migration

PVince81 · 2020-10-29T14:55:25Z

oc_dav_properties is using fileid to map DAV properties so that they can survive renames

Given that oc_properties only contains the path, DAV properties will likely get lost when renaming through some code paths.
see owncloud/core#25563

Maybe to keep oc_properties we could put a file id into the "propertiespath" column or add a new fileid (or objectid) column ?

Or in theory we could also discard (or keep for later) that table, in case whatever properties exist in there is anyway irrelevant for NC apps. Note that sometimes when Webdav is mounted as a Windows drive it will add a lot of Windows-specific properties. Deleting these shouldn't cause any issue as Windows would re-populate the next time.

juliusknorr · 2020-11-05T14:35:37Z

Thanks a lot for digging into this already :)

regarding bigints I think we should add these to NC (and not convert bigint to int in NC)

Yes, definitely. At least bigint conversions are fine for minor releases if we let the user do the migration through occ db:convert-filecache-bigint

not sure if such change would be backported to NC 20 (if DB changes are allowed in minor releases), so might only arrive with NC 21 then ?

@rullzer what do you think?

juliusknorr · 2020-11-27T14:50:37Z

Or in theory we could also discard (or keep for later) that table, in case whatever properties exist in there is anyway irrelevant for NC apps. Note that sometimes when Webdav is mounted as a Windows drive it will add a lot of Windows-specific properties. Deleting these shouldn't cause any issue as Windows would re-populate the next time.

I'd actually say we can go that way. We just recreate the table then with our format and mainly need to migrate oc_dav_properties from that to ours. For that migration I'm not entirely sure if there is any other relevant properties than calendar ones (cc @skjnldsv does store contacts some in there as well) we could just extract the user id from the propertypath, but my DAV internal knowledge is a bit limited in that regard. So any input welcome. Maybe @icewind1991 or @rullzer can also have a look if that makes sense.

For the ownCloud oc_dav_properties migration to oc_properties i would just try to match against /calendar/userid/ and extract the userid for our table format from that.

+----+------------------------+------------------------------------------+---------------+--------------+
| id | propertypath           | propertyname                             | propertyvalue | propertytype |
+----+------------------------+------------------------------------------+---------------+--------------+
|  1 | calendars/admin/sdfsdf | {http://owncloud.org/ns}calendar-enabled | 0             |            1 |
+----+------------------------+------------------------------------------+---------------+--------------+

becomes

+----+--------+--------------------------+------------------------------------------+---------------+
| id | userid | propertypath             | propertyname                             | propertyvalue |
+----+--------+--------------------------+------------------------------------------+---------------+
|  1 | admin  | calendars/admin/personal | {http://owncloud.org/ns}calendar-enabled | 0             |
+----+--------+--------------------------+------------------------------------------+---------------+

The propertytype introduced in owncloud/core#37314 is also most likely irrelevant for us for now.

rullzer · 2020-12-03T13:55:21Z

How did you get to that userid? shouldn't it be admin then?

juliusknorr · 2020-12-03T13:59:39Z

Woops, must have been a copy past error when trying to align copied dumps from the the two instances i had. Yes that would of course be admin.

core/Migrations/Version13000Date20170718121200.php

nickvergessen · 2020-12-07T18:43:01Z

core/Migrations/Version13000Date20170919121250.php

+			$column->setUnsigned(true);
+		} else {
+			$table->addColumn('remember', 'smallint', [
+				'notnull' => true,


invalid default

Fixed, there were actually some more columns affected by that in the authtoken table

Signed-off-by: Julius Härtl <jus@bitgrid.net>

This reverts commit d9b1492. Signed-off-by: Julius Härtl <jus@bitgrid.net>

Signed-off-by: Julius Härtl <jus@bitgrid.net>

nickvergessen · 2020-12-09T13:56:38Z

lib/private/Repair/Owncloud/MoveAvatarsBackgroundJob.php

+use OC\BackgroundJob\QueuedJob;
+use OCP\Files\IRootFolder;
+use OCP\Files\NotFoundException;
+use OCP\Files\Storage;
+use OCP\IAvatarManager;
+use OCP\ILogger;
+use OCP\IUser;
+use OCP\IUserManager;
+use Psr\Log\LoggerInterface;


something is unused

nickvergessen

🐘

as the files are not scanned we cannot use the OCP\Files api Signed-off-by: Julius Härtl <jus@bitgrid.net>

Signed-off-by: Julius Härtl <jus@bitgrid.net>

Signed-off-by: Julius Härtl <jus@bitgrid.net> Revert "Make sure the migrations table schema is always checked" This reverts commit 258955e. Set current vendor during upgrade and perform migrations table change if needed Signed-off-by: Julius Härtl <jus@bitgrid.net>

rullzer · 2020-12-10T20:33:27Z

/backport to stable20

PVince81 · 2020-12-22T13:28:16Z

seems we missed oc_share_external: #24813

PVince81 · 2020-12-22T13:38:17Z

core/Migrations/Version21000Date20201120141228.php

+
+		if ($schema->hasTable('systemtag')) {
+			$table = $schema->getTable('systemtag');
+			if ($table->hasColumn('systemtag')) {


wrong check, this should be "hasColumn('assignable')"

dossantg · 2021-01-12T14:58:09Z

Is it possible to backport this PR for 19.0.7? I think the commit (c77e259) for indice on oc_cards can fix #23980

But maybe i am wrong!

skjnldsv added 2. developing Work in progress enhancement high labels Oct 31, 2020

skjnldsv added this to the Nextcloud 21 milestone Oct 31, 2020

juliusknorr force-pushed the migration-10.5 branch from 3a3991b to 4425bf7 Compare November 20, 2020 15:17

nextcloud deleted a comment from faily-bot bot Nov 27, 2020

juliusknorr force-pushed the migration-10.5 branch 2 times, most recently from d24ec24 to 7520da6 Compare December 7, 2020 14:46

juliusknorr added 3. to review Waiting for reviews and removed 2. developing Work in progress labels Dec 7, 2020

juliusknorr marked this pull request as ready for review December 7, 2020 15:14

juliusknorr requested review from MorrisJobke, nickvergessen, rullzer and PVince81 and removed request for MorrisJobke and nickvergessen December 7, 2020 15:14

nickvergessen requested changes Dec 7, 2020

View reviewed changes

juliusknorr mentioned this pull request Dec 8, 2020

Use string for storing a OCM remote id #24247

Merged

juliusknorr added 9 commits December 9, 2020 12:13

Drop oc_dav_job_status table

de67719

Signed-off-by: Julius Härtl <jus@bitgrid.net>

Drop assignable systemtag column

f276ccb

Signed-off-by: Julius Härtl <jus@bitgrid.net>

Drop attributes on oc_share table

fe80a65

Signed-off-by: Julius Härtl <jus@bitgrid.net>

Add missing mount_id index

76addfa

Signed-off-by: Julius Härtl <jus@bitgrid.net>

Move authtoken login_name column to varchar(255)

3c9218a

Signed-off-by: Julius Härtl <jus@bitgrid.net>

Add missing index on oc_cards and rename if it previously existed

c77e259

Signed-off-by: Julius Härtl <jus@bitgrid.net>

Handle oc_dav_properties migration

28491ad

Signed-off-by: Julius Härtl <jus@bitgrid.net>

Readd repair steps that are relevant when migrating from ownCloud

9fe94f2

This reverts commit d9b1492. Signed-off-by: Julius Härtl <jus@bitgrid.net>

Dump autoloader

4ef148a

Signed-off-by: Julius Härtl <jus@bitgrid.net>

juliusknorr force-pushed the migration-10.5 branch from 353d6f3 to 82c3f16 Compare December 9, 2020 11:14

nextcloud deleted a comment from faily-bot bot Dec 9, 2020

nickvergessen reviewed Dec 9, 2020

View reviewed changes

nickvergessen approved these changes Dec 9, 2020

View reviewed changes

juliusknorr added 3 commits December 9, 2020 15:23

Properly migrate from new owncloud avatar location

20949d7

as the files are not scanned we cannot use the OCP\Files api Signed-off-by: Julius Härtl <jus@bitgrid.net>

Change further columns to be nullable with a default of 0

0befe07

Signed-off-by: Julius Härtl <jus@bitgrid.net>

juliusknorr force-pushed the migration-10.5 branch from f6da1c3 to 36ffad5 Compare December 9, 2020 14:24

rullzer approved these changes Dec 10, 2020

View reviewed changes

rullzer merged commit ccd5ca5 into master Dec 10, 2020

rullzer deleted the migration-10.5 branch December 10, 2020 13:32

backportbot-nextcloud bot added the backport-request label Dec 10, 2020

backportbot-nextcloud bot mentioned this pull request Dec 10, 2020

[stable20] Handle owncloud migration to latest release #24653

Merged

backportbot-nextcloud bot removed the backport-request label Dec 10, 2020

PVince81 reviewed Dec 22, 2020

View reviewed changes

PVince81 mentioned this pull request Dec 23, 2020

Missing columns after OC 10.5 -> NC 21.0.0beta2 upgrade #24813

Closed

1 task

nickvergessen mentioned this pull request Jan 7, 2021

Fix column name to check prior to deleting #25001

Merged

SystemKeeper mentioned this pull request Dec 9, 2023

Correct the version of owncloud allowing migration nextcloud/documentation#11374

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Handle owncloud migration to latest release #23044

Handle owncloud migration to latest release #23044

juliusknorr commented Sep 25, 2020 •

edited by PVince81

Loading

PVince81 commented Oct 29, 2020

PVince81 commented Oct 29, 2020

PVince81 commented Oct 29, 2020

PVince81 commented Oct 29, 2020

PVince81 commented Oct 29, 2020 •

edited

Loading

juliusknorr commented Nov 5, 2020

juliusknorr commented Nov 27, 2020 •

edited

Loading

rullzer commented Dec 3, 2020

juliusknorr commented Dec 3, 2020

nickvergessen Dec 7, 2020

juliusknorr Dec 9, 2020

nickvergessen Dec 9, 2020

nickvergessen left a comment

rullzer commented Dec 10, 2020

PVince81 commented Dec 22, 2020

PVince81 Dec 22, 2020

dossantg commented Jan 12, 2021

Handle owncloud migration to latest release #23044

Handle owncloud migration to latest release #23044

Conversation

juliusknorr commented Sep 25, 2020 • edited by PVince81 Loading

Adjustments

Missing from the database diff between 10.5 and 20

Removing data from database (core/Migrations/Version21000Date20201120141228.php)

Pulled out

PVince81 commented Oct 29, 2020

PVince81 commented Oct 29, 2020

PVince81 commented Oct 29, 2020

PVince81 commented Oct 29, 2020

PVince81 commented Oct 29, 2020 • edited Loading

juliusknorr commented Nov 5, 2020

juliusknorr commented Nov 27, 2020 • edited Loading

rullzer commented Dec 3, 2020

juliusknorr commented Dec 3, 2020

nickvergessen Dec 7, 2020

Choose a reason for hiding this comment

juliusknorr Dec 9, 2020

Choose a reason for hiding this comment

nickvergessen Dec 9, 2020

Choose a reason for hiding this comment

nickvergessen left a comment

Choose a reason for hiding this comment

rullzer commented Dec 10, 2020

PVince81 commented Dec 22, 2020

PVince81 Dec 22, 2020

Choose a reason for hiding this comment

dossantg commented Jan 12, 2021

juliusknorr commented Sep 25, 2020 •

edited by PVince81

Loading

PVince81 commented Oct 29, 2020 •

edited

Loading

juliusknorr commented Nov 27, 2020 •

edited

Loading