0.11.0 release checklist

[patrickjahns]

Reasons

Full list of PRs is here https://github.com/owncloud/user_ldap/milestone/11?closed=1 but summarised below.

• Primarily, we moved sync login into core and out of LDAP Sync should happen in core #156 @jvillafanez ❌ 0.11.0 release checklist #174 (comment)
• Killed all the old ajax.php and moved to controllers convert ajax/* to Controllers #199
• Much improved acceptance and UI automated tests
• UUIDs are not trimmed do not trim uuid #142 @jvillafanez ❗️ 0.11.0 release checklist #174 (comment)
• JSON output for list config command allow json output format #185 @jvillafanez ✅ (0.11.0 release checklist #174 (comment))
• Make sure new encryption keys use password with LDAP correctly When using encryption, ldap users login for the first time have invalid private keys. #177 @jvillafanez ✅

QA

• Create release branch - @patrickjahns - https://github.com/owncloud/user_ldap/tree/release/0.11.0
• Create "$version Testplan" @voroyam 0.11.0 Release Test #230
• Test and document in testplan => Final "Go" from QA (set 'QA-ready' label)

Marketing

• Check/adapt info.xml stuff
• Check/adapt screenshot
• Blog post or other communication if applicable

Documentation

• Changes in the docs needed?

Build & Marketplace release

• Bump version in info.xml
• Update CHANGELOG.md - Change CHANGELOG.md 'Unreleased' to '$version' @patrickjahns
• Create github tag for '$version' @patrickjahns
• Build and sign tarball from github tag '$version' @patrickjahns
• Upload to marketplace @patrickjahns
• Send mail to release-coordination with link to this ticket and to the changelog @patrickjahns
• Rename QA milestone to '$version' @patrickjahns
• Add a new 'Unreleased' section in CHANGELOG.md @patrickjahns
• Create PR to merge changes back to master @patrickjahns

[patrickjahns]

Pre-release tarball available at https://cloud.owncloud.com/index.php/s/Iuu1tBK997GxqLE

NOTE: this tarball is not signed as it is pre-release

[jvillafanez]

• BUG single user syncing considers the user as missing:

  1. Setup ownCloud with user_ldap
  2. Run occ user:sync "OCA\User_LDAP\User_Proxy" to sync all the users
  3. Run occ user:sync -u <uid> "OCA\User_LDAP\User_Proxy"

  root@e735b7973972:/opt/owncloud# sudo -u www-data ./occ user:sync -u 9c7b4656-d008-1036-8555-dd0c704ad877 "OCA\User_LDAP\User_Proxy"If unknown users are found, what do you want to do with their accounts? (removing the account will also remove its data)
    [0] disable
    [1] remove
    [2] ask later
   > 2
  Syncing 9c7b4656-d008-1036-8555-dd0c704ad877 ...
  These accounts that are no longer available in the backend:
  9c7b4656-d008-1036-8555-dd0c704ad877
  What do you want to do with their accounts? (removing the account will also remove its data)
    [0] disable
    [1] remove
   > 0
  Disabling accounts
  9c7b4656-d008-1036-8555-dd0c704ad877, aaliyah aañote, aannote@pmail.com disabled
  

• BUG --showCount doesn't seem to work properly (no difference with the option set regarding normal syncing) Didn't notice the "x/y" nor the percentage while inserting. It might need a better help message

  root@e735b7973972:/opt/owncloud# sudo -u www-data ./occ user:sync -c "OCA\User_LDAP\User_Proxy"
  If unknown users are found, what do you want to do with their accounts? (removing the account will also remove its data)
    [0] disable
    [1] remove
    [2] ask later
   > 2
  Analysing known accounts ...
   1002 [============================]
  
  No removed users have been detected.
  
  
  Inserting new and updating all known users from OCA\User_LDAP\User_Proxy ...
   1001/1001 [============================] 100%
  

• BUG --seen-only doesn't sync properly with the default setup:

  mysql> select * from accounts where last_login <> 0;
  +----+-------+--------------------------------------+--------------------------------------+--------------+-------+------------+--------------------------+---------------------------------------------------------+-------+
  | id | email | user_id                              | lower_user_id                        | display_name | quota | last_login | backend                  | home                                                    | state |
  +----+-------+--------------------------------------+--------------------------------------+--------------+-------+------------+--------------------------+---------------------------------------------------------+-------+
  |  1 | NULL  | admin                                | admin                                | admin        | NULL  | 1524048174 | OC\User\Database         | /opt/owncloud/data/admin                                |     1 |
  |  2 | NULL  | 9c8a6672-d008-1036-8584-dd0c704ad877 | 9c8a6672-d008-1036-8584-dd0c704ad877 | 0            | NULL  | 1524060844 | OCA\User_LDAP\User_Proxy | /opt/owncloud/data/9c8a6672-d008-1036-8584-dd0c704ad877 |     1 |
  |  3 | NULL  | 9c7b4656-d008-1036-8555-dd0c704ad877 | 9c7b4656-d008-1036-8555-dd0c704ad877 | 0            | NULL  | 1524060825 | OCA\User_LDAP\User_Proxy | /opt/owncloud/data/9c7b4656-d008-1036-8555-dd0c704ad877 |     1 |
  |  5 | NULL  | 9c7efba2-d008-1036-8559-dd0c704ad877 | 9c7efba2-d008-1036-8559-dd0c704ad877 | 0            | NULL  | 1524061108 | OCA\User_LDAP\User_Proxy | /opt/owncloud/data/9c7efba2-d008-1036-8559-dd0c704ad877 |     1 |
  +----+-------+--------------------------------------+--------------------------------------+--------------+-------+------------+--------------------------+---------------------------------------------------------+-------+
  4 rows in set (0.00 sec)
  

  Expected (fixed after login):

  mysql> select * from accounts where last_login <> 0;
  +----+---------------------+--------------------------------------+--------------------------------------+-----------------------+-------+------------+--------------------------+---------------------------------------------------------+-------+
  | id | email               | user_id                              | lower_user_id                        | display_name          | quota | last_login | backend                  | home                                                    | state |
  +----+---------------------+--------------------------------------+--------------------------------------+-----------------------+-------+------------+--------------------------+---------------------------------------------------------+-------+
  |  1 | NULL                | admin                                | admin                                | admin                 | NULL  | 1524048174 | OC\User\Database         | /opt/owncloud/data/admin                                |     1 |
  |  2 | brown@ex.com        | 9c8a6672-d008-1036-8584-dd0c704ad877 | 9c8a6672-d008-1036-8584-dd0c704ad877 | aaliyah brown         | NULL  | 1524061541 | OCA\User_LDAP\User_Proxy | /opt/owncloud/data/9c8a6672-d008-1036-8584-dd0c704ad877 |     1 |
  |  3 | aannote@example.com | 9c7b4656-d008-1036-8555-dd0c704ad877 | 9c7b4656-d008-1036-8555-dd0c704ad877 | aaliyah aañote        | NULL  | 1524061575 | OCA\User_LDAP\User_Proxy | /opt/owncloud/data/9c7b4656-d008-1036-8555-dd0c704ad877 |     1 |
  |  5 | NULL                | 9c7efba2-d008-1036-8559-dd0c704ad877 | 9c7efba2-d008-1036-8559-dd0c704ad877 | aaliyah abernathy (1) | NULL  | 1524061517 | OCA\User_LDAP\User_Proxy | /opt/owncloud/data/9c7efba2-d008-1036-8559-dd0c704ad877 |     1 |
  +----+---------------------+--------------------------------------+--------------------------------------+-----------------------+-------+------------+--------------------------+---------------------------------------------------------+-------+
  4 rows in set (0.00 sec)
  

Working:

• --re-enable
• --missing-account-action
• user sync during login

[jvillafanez]

#142 is untesteable. According to http://ldapwiki.com/wiki/EntryUUID the entryUUID value is automatically generated by the server and can't be modified by the user.
I've also tried to use the uid but I'm having a lot of problems trying to include a whitespace or newline at the beginning or end of the uid. I'm not sure if it's due to the uid being part of the dn. Other attributes will likely cause trouble.

[jvillafanez]

#185 looks fine, although with multiple ldap configuration it returns multiple jsons. I'd rather show only one json with multiple entries such as

{
  "": {"ldapHost":"....", "ldapPort":"2222", ....},
  "s01": {"ldapHost":...", "ldapPort":"4444", ....}
}

Same format for just one configuration.

[butonic]

@jvillafanez you were testing with AD? and objectguid was in the additional search attributes?

[jvillafanez]

It was openLDAP with the default configuration, just added the attributes for the quota and mail (group membership also changed to "member"). No additional search attributes were set.

[butonic]

Testing against samba4 as AD DC:

➜  core git:(stable10) ✗ ./occ ldap:show-config                                                                            
+-------------------------------+-------------------------------------------------------------------------------------------+
| Configuration                 | s01                                                                                       |
+-------------------------------+-------------------------------------------------------------------------------------------+
| hasMemberOfFilterSupport      | 0                                                                                         |
| hasPagedResultSupport         |                                                                                           |
| homeFolderNamingRule          |                                                                                           |
| lastJpegPhotoLookup           | 0                                                                                         |
| ldapAgentName                 | cn=Administrator,cn=users,dc=home,dc=oberdreyer,dc=de                                     |
| ldapAgentPassword             | ***                                                                                       |
| ldapAttributesForGroupSearch  |                                                                                           |
| ldapAttributesForUserSearch   | objectguid                                                                                |
| ldapBackupHost                |                                                                                           |
| ldapBackupPort                |                                                                                           |
| ldapBase                      | dc=home,dc=oberdreyer,dc=de                                                               |
| ldapBaseGroups                | cn=groups,dc=home,dc=oberdreyer,dc=de                                                     |
| ldapBaseUsers                 | cn=users,dc=home,dc=oberdreyer,dc=de                                                      |
| ldapCacheTTL                  | 600                                                                                       |
| ldapConfigurationActive       | 1                                                                                         |
| ldapDynamicGroupMemberURL     |                                                                                           |
| ldapEmailAttribute            | mail                                                                                      |
| ldapExperiencedAdmin          | 0                                                                                         |
| ldapExpertUUIDGroupAttr       |                                                                                           |
| ldapExpertUUIDUserAttr        | objectguid                                                                                |
| ldapExpertUsernameAttr        |                                                                                           |
| ldapGroupDisplayName          | cn                                                                                        |
| ldapGroupFilter               | (&(|(objectclass=group)))                                                                 |
| ldapGroupFilterGroups         |                                                                                           |
| ldapGroupFilterMode           | 1                                                                                         |
| ldapGroupFilterObjectclass    | group                                                                                     |
| ldapGroupMemberAssocAttr      | member                                                                                    |
| ldapHost                      | 127.0.0.1                                                                                 |
| ldapIgnoreNamingRules         |                                                                                           |
| ldapLoginFilter               | (&(|(objectclass=person))(|(samaccountname=%uid)(|(mailPrimaryAddress=%uid)(mail=%uid)))) |
| ldapLoginFilterAttributes     |                                                                                           |
| ldapLoginFilterEmail          | 1                                                                                         |
| ldapLoginFilterMode           | 0                                                                                         |
| ldapLoginFilterUsername       | 1                                                                                         |
| ldapNestedGroups              | 0                                                                                         |
| ldapOverrideMainServer        |                                                                                           |
| ldapPagingSize                | 500                                                                                       |
| ldapPort                      | 389                                                                                       |
| ldapQuotaAttribute            |                                                                                           |
| ldapQuotaDefault              |                                                                                           |
| ldapTLS                       | 0                                                                                         |
| ldapUserDisplayName           | cn                                                                                        |
| ldapUserDisplayName2          |                                                                                           |
| ldapUserFilter                | (|(objectclass=person))                                                                   |
| ldapUserFilterGroups          |                                                                                           |
| ldapUserFilterMode            | 1                                                                                         |
| ldapUserFilterObjectclass     | person                                                                                    |
| ldapUuidGroupAttribute        | auto                                                                                      |
| ldapUuidUserAttribute         | auto                                                                                      |
| turnOffCertCheck              | 0                                                                                         |
| useMemberOfToDetectMembership | 1                                                                                         |
+-------------------------------+-------------------------------------------------------------------------------------------+
➜  core git:(stable10) ✗ ./occ user:sync "OCA\User_LDAP\User_Proxy" -m disable -vvv -u 1415264A-5842-45F6-897F-E8B26CF68104
Syncing 1415264A-5842-45F6-897F-E8B26CF68104 ...

➜  core git:(stable10) ✗ ./occ:user delete 1415264A-5842-45F6-897F-E8B26CF68104
zsh: datei oder Verzeichnis nicht gefunden: ./occ:user
➜  core git:(stable10) ✗ ./occ user:delete 1415264A-5842-45F6-897F-E8B26CF68104
User with uid '1415264A-5842-45F6-897F-E8B26CF68104', display name 'jfd', email '' was deleted
➜  core git:(stable10) ✗ ./occ user:sync "OCA\User_LDAP\User_Proxy" -m disable -vvv -u 1415264A-5842-45F6-897F-E8B26CF68104
Syncing 1415264A-5842-45F6-897F-E8B26CF68104 ...

➜  core git:(stable10) ✗ ./occ user:sync "OCA\User_LDAP\User_Proxy" -m disable -vvv                                        
Analysing known accounts ...
    7 [============================] < 1 sec 20.0 MiB

No removed users have been detected.


Inserting new and updating all known users from OCA\User_LDAP\User_Proxy ...
    5 [============================] < 1 sec 20.0 MiB

➜  core git:(stable10) ✗ ./occ user:sync "OCA\User_LDAP\User_Proxy" -m disable -vvv -u 1415264A-5842-45F6-897F-E8B26CF68104
Syncing 1415264A-5842-45F6-897F-E8B26CF68104 ...

@jvillafanez for openldap add entryuuid to the search attributes.

[jvillafanez]

My point is that it's unlikely that people will include that attribute unless told, and there is no warning message that tell the user that he need to include the attribute. This implies that normal users will open tickets feeling that the behaviour is buggy because the command doesn't do what they expect.

I'm not sure if it's easy to include the attribute automatically or transparently without user intervention, but it's something we have to try.

[butonic]

We could just include the configured uuid attribute in the user search.

But what we really need is more control over how we want to search. For user sync the uuid attribute alone should be used. there are cases when all uuids change (export, mass change, re import). Then an admin might want to sync based on samaccountname.

For that we first need to agree what username, login, id, userid, uuid and email mean when talking about users. If we can agree that

• login = the string that was used to login in the login page or the saml token. can be the username, email or eg the userPrincipalName, can be reassigned, is unique at a given point in time, case insensitive
• username = username shown to other users, can be reassigned, is unique at a given point in time, case insensitive
• email = email address, can be reassigned, the primary email is unique at a given point in time (currently not the case for our account table), case insensitive
• id = numeric internal id, used for foreign keys inside oc
• userid = the userid, not reassignable, unique, should be a uuid, but sometimes another generated id is preferred. should not be samaccount or userPrincipalName because they are reassignable and may not be unique, may be used to reference the user when talking to other systems
• uuid = an RFC uuid, eg entryuuid or objectguid ... which are generated automatically for all objects in every ldap server
• displayname = readable string
• cloudId = @domain.tld
• cloudUserName = @domain.tld, unique at a given point in time
• cloudDisplayName = ?

see owncloud/core#30617 and #223

then to REALLY fix this:

1. [WIP] properly name account uuid and username columns core#30617
2. allow array of attribute=>searchfilter in getUsers instead of simple search string
3. use the new api in core to search for a specific uuid with eg ['uid'=>'5e84ba3f-0ac8-419c-8512-5dc8a4f02758']
4. make the user backends aware of the new option

actually it is more complicated. we need to introduce a new API in core that apps can implement, because the existing API cannot be changed. If we sent an array or a weird string in the getUsers($pattern) argument old apps would choke, because they expect a string.

maybe we can implement a search object that implements __toString() as a fallback?