Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Scanner doesn't propagate etags any more when file changed #24255

Closed
PVince81 opened this issue Apr 25, 2016 · 14 comments · Fixed by #24256
Closed

Scanner doesn't propagate etags any more when file changed #24255

PVince81 opened this issue Apr 25, 2016 · 14 comments · Fixed by #24256
Assignees
@icewind1991
Labels
comp:filesystem regression sev1-critical Type:Bug
Milestone
9.1

Comments

@PVince81
Copy link
Contributor

Steps:

  1. Pick an external storage type (SFTP or SMB)
  2. Directly on the storage, create a folder "updatetest/sub/test.txt"
  3. Setup that external storage in ownCloud
  4. Close the web browser to avoid interferences (buildstoragestats, etc)
  5. Run occ files:scan --all
  6. Check database: select fileid,path,size,mtime,storage_mtime,etag from oc_filecache where storage=3 order by path;. Take note of etags.
  7. Change the contents of the file "test.txt" directly on the storage, not through ownCloud.
  8. Run occ files:scan --all
  9. Check database: select fileid,path,size,mtime,storage_mtime,etag from oc_filecache where storage=3 order by path;. Compare etags.

Expected result

After the second scan, the etag and size of all parents were updated up to the storage root.

Actual result

After the second scan, only the size changed but not the etag.
Only the etag of "test.txt" changed, and so does its "storage_mtime".

Versions

Works in ownCloud 8.2.3
Broken in ownCloud 9.0.1

@icewind1991 please help fixing this regression, thanks.

CC @MorrisJobke @bboule
@PVince81 PVince81 added this to the 9.0.2-current-maintenance milestone Apr 25, 2016
@PVince81
Copy link
Contributor Author

This is critical because it's the only way to properly make ownCloud detect remote changes for now.

@icewind1991
Copy link
Contributor

can reproduce

@icewind1991 icewind1991 mentioned this issue Apr 25, 2016
Merged
@icewind1991
Copy link
Contributor

fix is here #24256

@PVince81
Copy link
Contributor Author

@icewind1991 thanks a lot. Any reason why it wasn't triggered any more ? Or did it happen indirectly through the recent changes to propagation code ?

@ghost
Copy link

ghost commented Apr 25, 2016

@PVince81 Could it be possible that this is also fixing #23019 ?

@PVince81
Copy link
Contributor Author

@RealRancor unlikely.

@ghost
Copy link

ghost commented Apr 25, 2016

Mhh, but the user there has issues with oC9 that files changed on external storages are not synced / detected by the sync client. And shouldn't this be caused if etags are not updated / correclty propagated?

@PVince81
Copy link
Contributor Author

PVince81 commented Apr 25, 2016 •
edited
Loading

Ah, that part yes. I only saw "out of memory" issues, which won't be solved by this PR.

@PVince81 PVince81 mentioned this issue Apr 25, 2016
Closed
@MorrisJobke MorrisJobke modified the milestones: 9.1-current, 9.0.2-current-maintenance Apr 26, 2016
@Kaelber
Copy link

Kaelber commented May 4, 2016

Update :

with oc 9.0.2 RC-2 and oc_9.1.0 pAlp-1 and Client 2.2.0 Beta 2 the same situation :

Synology DS414 mit DSM 5.2 | ? : rescan runs and logs are storing, but no file update is running from server to client.
Synology DS712+ mit DSM 6 | ? : rescan runs and logs are storing, but no file update is running from server to client.

Regarding the filescan I got from the Logfile as follow :

With oc 9.0.1 (not 9.1.0)

Starting scan for user 1 out of 1 (rescan)

+---------+-------+--------------+
| Folders | Files | Elapsed time |
+---------+-------+--------------+
| 12706 | 52211 | 01:04:55 |
+---------+-------+--------------+

----- End ----------

with oc 9.0.2 RC-2

Starting scan for user 1 out of 1 (rescan)

----- End ----------

@PVince81
Copy link
Contributor Author

PVince81 commented May 4, 2016

@Kaelber can you check the etags like I did in my original steps ?

@Kaelber
Copy link

Kaelber commented May 9, 2016 •
edited by PVince81
Loading

To test I got the problem with oc 9.0.2 from the filescan log as follows :

An unhandled exception has been thrown:
exception 'Doctrine\DBAL\DBALException' with message 'Failed to connect to the database: An exception occured in driver: could not find driver' in /volume1/web/owncloud/lib/private/db/connection.php:54
Stack trace:

#0 /volume1/web/owncloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(429): OC\DB\Connection->connect()
#1 /volume1/web/owncloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(389): Doctrine\DBAL\Connection->getDatabasePlatformVersion()
#2 /volume1/web/owncloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(328): Doctrine\DBAL\Connection->detectDatabasePlatform()
#3 /volume1/web/owncloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(621): Doctrine\DBAL\Connection->getDatabasePlatform()
#4 /volume1/web/owncloud/lib/private/db/connection.php(137): Doctrine\DBAL\Connection->setTransactionIsolation(2)
#5 /volume1/web/owncloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/DriverManager.php(172): OC\DB\Connection->__construct(Array, Object(Doctrine\DBAL\Driver\PDOMySql\Driver), Object(Doctrine\DBAL\Configuration), Object(Doctrine\Common\EventManager))
#6 /volume1/web/owncloud/lib/private/db/connectionfactory.php(118): Doctrine\DBAL\DriverManager::getConnection(Array, Object(Doctrine\DBAL\Configuration), Object(Doctrine\Common\EventManager))
#7 /volume1/web/owncloud/lib/private/server.php(366): OC\DB\ConnectionFactory->getConnection('mysql', Array)
#8 /volume1/web/owncloud/3rdparty/pimple/pimple/src/Pimple/Container.php(113): OC\Server->OC\{closure}(Object(OC\Server))
#9 /volume1/web/owncloud/lib/private/appframework/utility/simplecontainer.php(102): Pimple\Container->offsetGet('DatabaseConnect...')
#10 /volume1/web/owncloud/lib/private/servercontainer.php(87): OC\AppFramework\Utility\SimpleContainer->query('DatabaseConnect...')
#11 /volume1/web/owncloud/lib/private/server.php(886): OC\ServerContainer->query('DatabaseConnect...')
#12 /volume1/web/owncloud/lib/private/server.php(260): OC\Server->getDatabaseConnection()
#13 /volume1/web/owncloud/3rdparty/pimple/pimple/src/Pimple/Container.php(113): OC\Server->OC\{closure}(Object(OC\Server))
#14 /volume1/web/owncloud/lib/private/appframework/utility/simplecontainer.php(102): Pimple\Container->offsetGet('AppConfig')
#15 /volume1/web/owncloud/lib/private/servercontainer.php(87): OC\AppFramework\Utility\SimpleContainer->query('AppConfig')
#16 /volume1/web/owncloud/lib/private/server.php(825): OC\ServerContainer->query('AppConfig')
#17 /volume1/web/owncloud/lib/private/server.php(411): OC\Server->getAppConfig()
#18 /volume1/web/owncloud/3rdparty/pimple/pimple/src/Pimple/Container.php(113): OC\Server->OC\{closure}(Object(OC\Server))
#19 /volume1/web/owncloud/lib/private/appframework/utility/simplecontainer.php(102): Pimple\Container->offsetGet('AppManager')
#20 /volume1/web/owncloud/lib/private/servercontainer.php(87): OC\AppFramework\Utility\SimpleContainer->query('AppManager')
#21 /volume1/web/owncloud/lib/private/server.php(1063): OC\ServerContainer->query('AppManager')
#22 /volume1/web/owncloud/lib/private/app.php(254): OC\Server->getAppManager()
#23 /volume1/web/owncloud/lib/private/app.php(103): OC_App::getEnabledApps()
#24 /volume1/web/owncloud/lib/base.php(577): OC_App::loadApps(Array)
#25 /volume1/web/owncloud/lib/base.php(1112): OC::init()
#26 /volume1/web/owncloud/console.php(46): require_once('/volume1/web/ow...')
#27 /volume1/web/owncloud/occ(11): require_once('/volume1/web/ow...')
#28 {main}

@PVince81
Copy link
Contributor Author

PVince81 commented May 9, 2016

@Kaelber it looks like you're having environment problems with your database

@Kaelber
Copy link

Kaelber commented May 30, 2016

Update :

with oc 9.1.0 Beta 1 and Client 2.2.0 the follow situation :

Synology DS414 mit DSM 5.2 | ? : scan runs and logs are storing, file only updating, if I use --all in the scan command. A USER in the command is NOT RUNNING, no update is done.

Synology DS712+ mit DSM 6 | ? : rescan runs and logs are storing, but no file update is running from server to client, also not with --all in the scan command

Regarding the filescan I got from the Logfile as follow :

Synology DS414 with DSM 5.2

Scanning files for 4 users
Starting scan for user 1 out of 4 (admin)
Starting scan for user 2 out of 4 (kaelber)
Starting scan for user 3 out of 4 (rescan)
Starting scan for user 4 out of 4 (kaelber2)

+---------+--------+--------------+
| Folders | Files | Elapsed time |
+---------+--------+--------------+
| 54235 | 238428 | 03:05:59 |
+---------+--------+--------------+

----- End ----------

Synology DS712+ with DSM 6

Scanning files for 4 users
Starting scan for user 1 out of 4 (admin)
Starting scan for user 2 out of 4 (kaelber)

----- End ----------

@lock
Copy link

lock bot commented Aug 4, 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked as resolved and limited conversation to collaborators Aug 4, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@icewind1991 icewind1991

Labels
comp:filesystem regression sev1-critical Type:Bug
Projects
None yet
Milestone
9.1
Development

Successfully merging a pull request may close this issue.

triger the propagator from the command line scanner owncloud/core
4 participants
@MorrisJobke @PVince81 @icewind1991 @Kaelber