Heavy performance drop of the user:sync command after one day

[jvillafanez]

Steps to reproduce

1. configure LDAP app to connect to one server
2. use time sudo -u www-data ./occ user:sync "OCA\User_LDAP\User_Proxy" -m remove and check how much time it takes to sync
3. wait one day (*)
4. repeat step 2

(*) You can go to https://github.com/owncloud/user_ldap/blob/master/lib/User/User.php#L361 and modify the interval to a more reasonable value for testing, maybe 300 or 600 secs. Then wait that time.

Expected behaviour

It takes a similar amount of time to sync the users, specially if there hasn't been any change in LDAP

Actual behaviour

The amount of time varies a lot depending on the information we need to retrieve from LDAP (quota, mail, search terms...) and whether we need to update information for LDAP.

Default configuration within the update interval

root@1176ad233fd3:/opt/owncloud# time sudo -u www-data ./occ user:sync "OCA\User_LDAP\User_Proxy" -m remove
Analyse unknown users ...
 1006 [============================]

No unknown users have been detected.
Insert new and update existing users ...
 1003/1003 [============================] 100%


real	0m14.058s
user	0m3.760s
sys	0m1.024s

Default configuration not within the update interval, without any change in the LDAP server

root@1176ad233fd3:/opt/owncloud# time sudo -u www-data ./occ user:sync "OCA\User_LDAP\User_Proxy" -m remove
Analyse unknown users ...
 1006 [============================]

No unknown users have been detected.
Insert new and update existing users ...
 1003/1003 [============================] 100%


real	4m10.363s
user	3m23.852s
sys	0m3.800s

Configuration with quota, mail, home folder naming rule, and 3 custom search attributes, not within the update interval, without changes in the LDAP server

root@1176ad233fd3:/opt/owncloud# time sudo -u www-data ./occ user:sync "OCA\User_LDAP\User_Proxy" -m remove
Analyse unknown users ...
 1006 [============================]

No unknown users have been detected.
Insert new and update existing users ...
 1003/1003 [============================] 100%


real	20m1.609s
user	17m46.392s
sys	1m27.748s

Additional notes

• The speed at what the unknown users are analyzed decrease with the numbers. This suggest that there is something more that more requests being made because the speed isn't constant.
• Fetching the information of 500 users can be slow to the point it might seem the command is hung, specially if we need to fetch mail, quota, and more things. Inserting the data in the DB seems very fast in comparation.

[felixboehm]

@butonic @tomneedham Please share your thoughts on this

[butonic]

caused by updating quota or email properties, which trigger an update of the addressbook cards by the card dav backend, which triggers fatching avatars, which triggers fs initialization, which takes ages ... several PRs incoming, but still needs

• a new interface for avatar provisioning and probably
• a better fs for avatars
• no longer initialize the fs for every user because the avatars now reside in data/avatars instead of user homes

[felixboehm]

@DeepDiver1975 potential performance improvements here in core?

[tomneedham]

owncloud/core#28216
owncloud/core#28215
owncloud/core#28214
owncloud/core#28213
#109

Have improved performance significantly - but we can go further with a proper interface for avatars - tracked here owncloud/core#28428

[butonic]

I need to work on the ldap sync part for emails, this is related.