Merge from 4.3.0 (#759)

* fix post setup for scim

* Casa's script for DUO

* oxBiometricDevices, oxDUODevices

* Putting oxDuoDevices and oxBiometricDevices at the bottom

* oxBiometricDevices and oxDuoDevices are a part of gluuCustomPerson and not GluuPerson

* fix custom_schema.json

* change desc in custom_schema.json

* (4.2.2) setup: revoke interception script sample

GluuFederation/oxAuth#1502

* ask if oxtrust to be installed

* couchase user pfrefix in datasource

* fix version in post-setup

* Version 4.2.2.Final

* Version 4.2.3-SNAPSHOT

* (4.2.2) setup: added ou=stat,o=gluu

GluuFederation/oxAuth#1512

* (4.2.3) setup: added jansStatEntry OC

GluuFederation/oxAuth#1512

* (4.2.3) setup: added jansId to indexes

GluuFederation/oxAuth#1512

* change mod_ssl name for rhel7

* data type imapdata -> json for attrib 42E1

* ldap2cb: migrate to py3 and fixes

* Make some SCIM attributes multivalued as in spec

* Add multivalued data

* Version 4.2.3.Final

* (4.2.3) setup: added statWebServiceIntervalLimitInSeconds:60

GluuFederation/oxAuth#1512

* mod_ssl centos7

* Version 4.3.0.Final

* Add u2f and fido2 test data

* Add sample update_token script

* Add sample update_token script

* fixes templates

* show version on tui

* fixes

* no-chroot install script

* install missing packages

* Add keepAliveInterval CB SDK support

* F2 to display version info

* python3-six dependency for gluu_setup.py

* refactor: 4.3.0 setup

* fix: oxd-server.default

* fix: idp download

* fix: don't backup same file

* refactor: seperate argparser

* refactor(rdbm): json files

* fix: add ou=stat,o=gluu

* feat: migrate Jans to Gluu OC

* feat: migrate Jans to Gluu OC

* refactor: rdbm works

* feat(rdbm): installation

* fix: gluu_installer.py for args

* fix: radius installer

* fix(rdbm): local install

* fix(schema): add missing attributes

* fix(rdbm): schema and sql data types

* fix: gluuPerson

* fix: update package list

* fix: update installer for missing packages

* fix: ruamel module

* fix: add pylib path to encode.py

* refactor: re-order installers

* fix: encode.py

* fix: doc_id gluu --> _

* fix: remov unused file db_utils_org.py

* feat: add -n option to gluu_install.py

* fix: change ce setup branch

* feat: implement --no-progress

* fix: redhat8 installs

* feat: add -no-setup to installer

* fix: typo

* fix: enable opendj

* re-add stat service attrbiutes

* fix: generate schema

* fix: set jetty timeout 300

* fix: set systemd tiemout

* fix: spanner fixes

* feat: logging config

* fix: cb installation

* refactor: log filename db-backend.log

* feat: add --dist-server-base

* fix: couchbase test data loading

* fix: cn for uniqueness in ldap

* fix: load test data for ldap

* fix: test data loader ldap bind

* fix: dsconfig after test data

* fix: remove attrbiutes lifetime & salt

* fix: implement spanner test data loader

* fix: add del and exp to gluuPasswordResetRequest

* fix: re-generate schema

* feat: pre add base metric entries

* feat: pre add base metric entries

* feat: pre add base metric entries

* fix: spacing

* Update install.py

added sqlalchemy extraction to ces_dir

* fix: added ssnId to oxAuthUmaRPT

* fix: added ssnId to oxAuthUmaPCT

* fix: load ldif

* fix(wrends): change display text to opendj

* fix: spanner subtables

* fix: couchbase install (ref: #741)

* faet: review Spanner indexes #736

* fix: remote couchbase arg

* faet: review Spanner indexes #736

* fix: mysql indexes (ref: #737)

* fix: chown root:gluu gluuOptPythonFolder

* fix(spanner): passport installation (ref: #740)

* fix: creating o=metric related entries (ref: #742)

* fix: ldap test data loader

* fix: spanner test data loading

* feat(test-data): display if test data will be loaded

* feat(cert): download Apple WebAuthn Root CA

* fix: setting couchbase admin password

* fix: passport cert files mode

* feat: merge test config changes from jans

* fix: saml couchbase install (ref: #741)

* fix: check if apache module is availabe before enabling (ref: #741)

* feat: check user and group before adding

* fix: remove wrong error line for rendering ecnode script

* feat: check if opendj ports are free (ref: #743)

* fix: /etc/certs permissions

* feat: fix attribute names to conform Gluu shema

* feat: update test CIBA configuration

* feat: update test CIBA configuration

* fix: test data additional columns creation

* feat: fix configuration entry DN in server tests

* fix: limit lenght of description for indexing

* fix: set size 768 for description

* fix: spanner test data columns

* fix: cb test oxauth config

* feat: update default server profile

* feat: update default server profile

* feat: update default server profile

* fix: don't create gluuCustomPerson

* fix: typo

* feat: update default server profile

* feat: command line backend options

* fix: set couchbase host to hostname for local installation

* fix: disable ssl ofr cb test profile

* feat: merge scim properties from Jans

* fix: couchbase hostname

* fix: rdbm test data columns

* fix: local mysql installation

* fix: typo

* fix: prepend plus sign GluuFederation/casa#138

* feat(tui): implement backends

* fix: oxd server gluu storage config

* fix: typo

* fix: oxd-server progress string

* fix: remote cb install

* fix: remove tmp file

* fix: create UMA SCIM resource (ref: #744)

* fix: typo

* fix: double backup when inserting lines to file

* fix: scim-rp.jks goes to bot output and certs dir

* feat: file descriptor limits for systemd services (ref: #734 #745 )

* fix: post install tasks

* refactor(backend): disable rdbm

* feat: re-try download three times on fail

* Load module before config 

https://support.gluu.org/other/9790/bug-in-httpdconf/

* fix: passport certs ownerships

* fix: remove sqlalchemy related code from install.py

* fix: remove gluu_install.py if extracted within container

* fix: chenage jetty version

* fix: typo

* Updated marketing messages

Updated marketing messages

* Revert "refactor(backend): disable rdbm"

This reverts commit e204f11.

* fix: re-do commit 0fc53c8

* fix: re-do commit 1ff6a74

* refactor(rdbm): suppress rdbm options

* fix: scim istallation

* fix: ownership of webapps dir (ref: #746)

* feat(setup4.3): added stat scope

GluuFederation/oxAuth#1554

* refactor(4.3setup): renamed scope stat -> jans_stat

GluuFederation/oxAuth#1554

* fix: centos packages

* chore: sync scim script wrt oxexternal

* chore: remove comment

* feat: add enable war updates to gluu_install.py

* feat: extract sqlalchemy

* feat: gluu_install.py update oxd-server

* feat: dummy installation

* feat: added o to oxAuthClient

* fix: typo on help

* fix: update shibboleth idp custom script

* chore: adjust length casa attributes

* fix: don't create config for oxd

* feat: Touch ID as a fido2 device (platform authenticator)

* feat: Adjust protection mode handling in SCIM

See GluuFederation/scim#20

* feat: scim scopes (ref: #750)

* feat: UMA mode for SCIM (ref: #752)

* feat: dbUtils set config by dn

* fix: find oxd_host when collecting properties

* feat: fido installer add do_import arg

* fix: collect properties for cb backend

* fix: collect properties oxd_host

* feat: backup option for copyFile

* fix: collect properties casa for cb backend

* fix: set_configuration for cb

* feat: function determine_key_gen_path()

* feat: conform script wrt latest changes GluuFederation/scim#18

* feat: Update OpenDJ version

* feat: add jetty version

* feat: adjust template for new field, see GluuFederation/scim#22

* fix: fix oxDeviceData size

* fix: fic consent script api version

* feat: jetty-10 integration

* fix: jetty inifile

* fix: jetty inifile

* feat: jetty version is available with option -a

* fix: arg parser

* fix: spanner passport install

* fix: spanner related issues

* feat: gluu-utils

* fix: check City in TUI (ref: #754)

* fix: updated idp.properties for Shibboleth IDP install

* feat: Casa plugin for Stytch Credentials

* fix: use Final binaries

* feat: remove oxaut-rp installation

* fix: idp.session.slop entry in idp.properties prevents Shib IDP start

* fix: opendj version 4.4.12

* fix: re-enable encoding setup.properties

* fix: prevent casa client vanish

* fix: Fix wrong steps count in consent script

* fix: load setup.properties

* fix: update jetty version

* fix: mem calculation when setup.properties loaded

* fix: set opendj ram constraint

* fix: gluu-radius unable to start

* fix: gluu-radius failed to start due to incorrect user/group

* fix: ownership issue

* fix: radius init.d script

Co-authored-by: Mustafa Baser <mbaser@mail.com>
Co-authored-by: Madhumita <madhu@gluu.org>
Co-authored-by: YuriyZ <yzabrovarniy@gmail.com>
Co-authored-by: Jose <bonustrack310@gmail.com>
Co-authored-by: David <nikdavnik@gmail.com>
Co-authored-by: Ganesh <ganesh.sharma@worldiswelcome.com>
Co-authored-by: Mike Schwartz <mike@gluu.org>
Co-authored-by: Djeumen Rolain <uprightech@gmail.com>

install.py

@@ -1,9 +1,11 @@
 #!/usr/bin/python3
 
+import sys
+sys.path.append('/usr/lib/python3.6/gluu-packaged/')
+
 import site
 import re
 import glob
-import sys
 import os
 import subprocess
 import argparse
@@ -18,6 +20,7 @@
 
 run_time = time.strftime("%Y-%m-%d_%H-%M-%S")
 ces_dir = '/install/community-edition-setup'
+app_dir = '/opt/dist/app'
 
 parser = argparse.ArgumentParser(description="This script extracts community-edition-setup package and runs setup.py without arguments")
 parser.add_argument('-o', help="download latest package from github and override current community-edition-setup", action='store_true')
@@ -98,6 +101,10 @@
 if os.path.exists(post_setup):
     os.chmod(post_setup, 33261)
 
+gluu_install = '/install/community-edition-setup/gluu_install.py'
+if os.path.exists(gluu_install):
+    os.remove(gluu_install)
+
 if argsp.o:
     npy_download_link = 'https://github.com/npcole/npyscreen/archive/master.zip'
     result = requests.get(npy_download_link, allow_redirects=True)
@@ -115,11 +122,31 @@
         target_dir = '/tmp/npyscreen_tmp'
         npyzip.extractall(target_dir)
         npyzip.close()
-        
+
         shutil.copytree(
             os.path.join(target_dir, parent_dir, 'npyscreen'),
             dest_dir
             )
 
         shutil.rmtree(target_dir)
 
+print("Extracting sqlalchemy")
+sqlalchemy_fn = os.path.join(app_dir, 'sqlalchemy.zip')
+sqlalchemy_zip = zipfile.ZipFile(sqlalchemy_fn)
+sqlalchemy_parent_dir = sqlalchemy_zip.filelist[0].filename
+target_dir = '/tmp/sqlalchemy_tmp'
+
+if os.path.exists(target_dir):
+    shutil.rmtree(target_dir)
+
+sqlalchemy_zip.extractall(target_dir)
+sqlalchemy_zip.close()
+
+sqlalchemy_dir = os.path.join(ces_dir, 'setup_app/pylib/sqlalchemy')
+
+shutil.copytree(
+    os.path.join(target_dir, sqlalchemy_parent_dir, 'lib/sqlalchemy'),
+    sqlalchemy_dir
+    )
+
+shutil.rmtree(target_dir)

pylib/generate_properties.py

@@ -307,15 +307,13 @@ def generate_properties(as_dict=False):
             'oxauth':    ('installOxAuth', 0.3, 0.7),
             'identity':  ('installOxTrust', 0.2),
             'idp':       ('installSaml', 0.2),
-            'oxauth-rp': ('installOxAuthRP', 0.1),
             'passport':  ('installPassport', 0.1),
         }
     else:
         jetty_services = {
             'oxauth':    ('installOxAuth', 0.2, 0.7),
             'identity':  ('installOxTrust', 0.25),
             'idp':       ('installSaml', 0.25),
-            'oxauth-rp': ('installOxAuthRP', 0.1),
             'casa':      ('installCasa', 0.1),
             'passport':  ('installPassport', 0.1),
         }

pylib/gluu_utils.py

@@ -74,10 +74,11 @@ def get_os_type():
             if row:
                 if row[0] == 'ID':
                     os_type = row[1].lower()
-                    if os_type == 'rhel':
-                        os_type = 'redhat'
+                    if os_type in  ('rhel', 'redhat'):
+                        os_type = 'red'
                 elif row[0] == 'VERSION_ID':
                     os_version = row[1].split('.')[0]
+    print("Detected OS", os_type, os_version)
     return os_type, os_version
 
 def read_properties_file(fn):

pylib/messages.py

@@ -3,11 +3,10 @@ class msg:
     MAIN_label = "System Information"
     HostForm_label = "Gathering Information"
     ServicesForm_label = "Select Services to Install"
-    DBBackendForm_label = "Choose to Store in WrenDS"
+    DBBackendForm_label = "Backend Install Options"
     StorageSelectionForm_label = "Hybrid Storage Selection"
     InstallStepsForm_label = "Installing Gluu Server"
     DisplaySummaryForm_label = "Gluu Server Installation Summary"
-
     decription = "Use setup.py to configure your Gluu Server and to add initial data required for oxAuth and oxTrust to start. If setup.properties is found in this folder, these properties will automatically be used instead of the interactive setup."
 
     os_type_label = "Detected OS"
@@ -34,7 +33,8 @@ class msg:
     installOxAuthRP_label = "Install OxAuthRP" 
     installPassport_label = "Install Passport" 
     installGluuRadius_label = "Install Radius"
-    wrends_storages_label = "Store on WrenDS"
+    wrends_storage_selection_label = "Choose to Store in OpenDJ"
+    wrends_storages_label = "Store in OpenDJ"
     installing_label = "Installing"
     installOxd_label = "Install Oxd"
     installCasa_label = "Install Casa"
@@ -58,7 +58,7 @@ class msg:
     oxtrust_admin_password_label = "oxTrust Admin Password"
     oxtrust_admin_password_warning = "oxTrust Admin Password should be at least six characters"
     max_ram_int_warning = "Please enter and integer value for Max ram"
-    memory_warning = "WARINIG: You don't have enough memory to run Gluu CE properly with selected applications."
+    memory_warning = "WARINIG: You don't have enough memory to run Gluu CE properly with selected applications. Continue anyway?"
 
     exit_from_app = "Setup is exiting. %(reason)s"
     not_to_continue = "Since you don't want to continue."
@@ -74,28 +74,31 @@ class msg:
 
 
     ask_installHttpd = "Install Apache HTTPD Server"
+    ask_installOxTrust = "Install oxTrust Admin GUI"
     ask_installSaml = "Install Shibboleth SAML IDP"
     ask_installOxAuthRP  = "Install oxAuth RP"
     ask_installPassport  = "Install Passport"
     ask_installGluuRadius = "Install Gluu Radius"
     ask_installCasa = "Install Casa"
     ask_installOxd = "Install Oxd"
-    ask_wrends_install = "Install WrenDS"
+    ask_wrends_install = "Install OpenDJ"
     ask_installScimServer = "Install Scim Server"
     ask_installFido2 = "Install Fido2"
 
 
-    wrends_install_options = ["Don't Install","Install Locally","Use Remote WrenDS"]
+    wrends_install_options = ["Don't Install","Install Locally","Use Remote OpenDJ"]
     oxd_url_label = "oxd Server URL"
     install_oxd_or_url_warning = "Please either enter oxd Server URL or check Install Oxd"
     oxd_connection_error = "Can't connect to oxd-server with url {}. Reason: {}"
     oxd_ssl_cert_error = "Hostname of oxd ssl certificate is {} which does not match {} casa won't start properly"
 
     ask_cb_install = "Couchbase Installation"
     cb_install_options = ["Don't Install","Install Locally","Use Remote Couchbase"]
+    cb_not_available = "Couchbase package is not found in /opt/dist/couchbase. Please put and retry."
+    cb_bucket_rolese = "Please check user {} has roles {} on bucket(s) {}"
 
-    ask_use_gluu_storage_oxd = "By default oxd uses its own db. Do you want to use Gluu Storage for Oxd?"
-    ask_use_gluu_storage_oxd_title = "Use Gluu Storage for Oxd?"
+    ask_use_gluu_storage_oxd = "oxd includes a built-in h2 database for persistence. Do you want to change it to the Authorization Server's persistence mechanism instead (recommended for high load only)?"
+    ask_use_gluu_storage_oxd_title = "Use Authorization Server's persistence for oxd?"
 
     notify_select_backend = "Please select one of the backends either local install or remote" 
     weak_password = "Password for {} must be at least 6 characters and include one uppercase letter, one lowercase letter, one digit, and one special character."
@@ -108,10 +111,10 @@ class msg:
                    "Hostname: hostname of this server. Detected hostname will be provided.\n"
                    "Organization Name: ......")
 
-    installation_completed = "Gluu Server installation successful! Point your browser to https://{}"
-
+    installation_completed = "Gluu Server installation successful!"
+    installation_completed_oxtrsut =  "Point your browser to https://{}"
     installation_description_java = "Corretto is a build of the Open Java Development Kit (OpenJDK) with long-term support from Amazon. Corretto is certified using the Java Technical Compatibility Kit (TCK) to ensure it meets the Java SE standard."
-    installation_description_opendj = "WrenDS is an LDAPv3 compliant directory service, which has been developed for the Java platform, providing a high performance, highly available, and secure store for the identities managed by your organization."
+    installation_description_opendj = "OpenDJ is an LDAPv3 compliant directory service, which has been developed for the Java platform, providing a high performance, highly available, and secure store for the identities managed by your organization."
     installation_description_oxauth = "oxAuth is an open source OpenID Connect Provider (OP) and UMA Authorization Server (AS). The project also includes OpenID Connect Client code which can be used by websites to validate tokens."
     installation_description_oxtrust = "oxTrust is a Weld based web application for Gluu Server administration."
     installation_description_saml = "The Gluu Server acts as a SAML identity provider (IDP) to support outbound SAML single sign-on (SSO)."