Cannot deploy with AutoTLS #141

HirossxD · 2023-01-30T08:00:38Z

HirossxD
Jan 30, 2023

Unable to restart service cloudera-scm-server when deploying cluster with autotls

Hello, when I am deploying cluster without security: tls in definition.yml in both mgmt and basic cluster sections. and without tls=True in the inventory file. like it is mentioned in this documentation.
Without these and playbook tag autotls, cluster is deployed successfully, after that, manual autotls enablement is functional with both root and nonroot user

I have tried all mentioned above, with setting autotls user in this file

But I am always getting this error.

TASK [cloudera.cluster.autotls : Restart Cloudera Manager Server] **************
Wednesday 25 January 2023  08:16:52 +0000 (0:00:03.192)       0:12:32.683 *****
fatal: [myhost1.domain.com]: FAILED! => {"changed": false, "msg": "Unable to restart service cloudera-scm-server: Failed to restart cloudera-scm-server.service: Connection timed out\nSee system logs and 'systemctl status cloudera-scm-server.service' for details.\n"}

$ systemctl status cloudera-scm-server.service

● cloudera-scm-server.service - Cloudera CM Server Service
   Loaded: loaded (/usr/lib/systemd/system/cloudera-scm-server.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2023-01-25 09:13:39 CET; 15min ago
 Main PID: 60455 (java)
    Tasks: 109
   Memory: 2.5G
   CGroup: /system.slice/cloudera-scm-server.service
           └─60455 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el7_9.x86_64/bin/java -cp .:/usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-...

Jan 25 09:13:39 myhost1 systemd[1]: Starting Cloudera CM Server Service...
Jan 25 09:13:39 myhost1 systemd[1]: Started Cloudera CM Server Service.
Jan 25 09:13:39 myhost1 cm-server[60455]: JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el7_9.x86_64
Jan 25 09:13:39 myhost1 cm-server[60455]: OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
Jan 25 09:13:41 myhost1 cm-server[60455]: ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the con...n logging.
Jan 25 09:13:45 myhost1 cm-server[60455]: 09:13:45.471 [main] ERROR org.hibernate.engine.jdbc.spi.SqlExceptionHelper - ERROR: relation "cm_version" does not exist
Jan 25 09:13:45 myhost1 cm-server[60455]: Position: 21
Hint: Some lines were ellipsized, use -l to show in full.

Also checked logs from /var/log/cloudera-scm-server/cloudera-scm-server.log

2023-01-25 09:16:52,245 INFO scm-web-107:com.cloudera.cmf.command.GenerateCmcaCmdWork: Persisting new CMCA to database
2023-01-25 09:16:52,252 INFO scm-web-107:com.cloudera.cmf.command.GenerateCmcaCmdWork: Configuring CM to turn on Auto-TLS
2023-01-25 09:16:52,254 INFO scm-web-107:com.cloudera.cmf.command.GenerateCmcaCmdWork: Setting TLS configuration: AGENT_TLS
2023-01-25 09:16:52,259 INFO scm-web-107:com.cloudera.cmf.command.GenerateCmcaCmdWork: Setting TLS configuration: WEB_TLS
2023-01-25 09:16:52,261 INFO scm-web-107:com.cloudera.cmf.command.GenerateCmcaCmdWork: Setting TLS configuration: NEED_AGENT_VALIDATION
2023-01-25 09:16:52,263 INFO scm-web-107:com.cloudera.cmf.command.GenerateCmcaCmdWork: Setting TLS configuration: KEYSTORE_PATH
2023-01-25 09:16:52,265 INFO scm-web-107:com.cloudera.cmf.command.GenerateCmcaCmdWork: Setting TLS configuration: KEYSTORE_PASSWORD
2023-01-25 09:16:52,267 INFO scm-web-107:com.cloudera.cmf.command.GenerateCmcaCmdWork: Setting TLS configuration: TRUSTSTORE_PATH
2023-01-25 09:16:52,269 INFO scm-web-107:com.cloudera.cmf.command.GenerateCmcaCmdWork: Setting TLS configuration: TRUSTSTORE_PASSWORD
2023-01-25 09:16:52,271 INFO scm-web-107:com.cloudera.cmf.command.GenerateCmcaCmdWork: Setting TLS configuration: HOST_CERT_GENERATOR
2023-01-25 09:16:52,274 INFO scm-web-107:com.cloudera.cmf.command.GenerateCmcaCmdWork: Setting TLS configuration: SSL_CERTIFICATE_HOSTNAME
2023-01-25 09:16:52,276 INFO scm-web-107:com.cloudera.cmf.command.GenerateCmcaCmdWork: Setting TLS configuration: AUTO_TLS_KEYSTORE_PASSWORD
2023-01-25 09:16:52,278 INFO scm-web-107:com.cloudera.cmf.command.GenerateCmcaCmdWork: Setting TLS configuration: AUTO_TLS_TRUSTSTORE_PASSWORD
2023-01-25 09:16:52,280 INFO scm-web-107:com.cloudera.cmf.command.GenerateCmcaCmdWork: Setting TLS configuration: AUTO_TLS_TYPE
2023-01-25 09:16:52,282 INFO scm-web-107:com.cloudera.cmf.command.flow.CmdStep: Executing command 1546333793 work: Configure the services on this cluster for Auto-TLS.
2023-01-25 09:16:52,282 INFO scm-web-107:com.cloudera.cmf.command.ConfigureAutoTlsServicesCmdWork: Configuring existing services to use Auto-TLS
2023-01-25 09:16:52,284 INFO scm-web-107:com.cloudera.cmf.model.DbCommand: Command 1546333793(GenerateCMCACommand) has completed. finalstate:FINISHED, success:true, msg:Suc
cessfully generated CMCA and enabled Auto-TLS
2023-01-25 09:16:52,286 INFO scm-web-107:com.cloudera.cmf.service.ServiceHandlerRegistry: Global Command GenerateCMCACommand launched with id=1546333793
2023-01-25 09:16:52,347 INFO scm-web-107:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing Global command ProcessStalenessCheckCommand BasicCmdArgs{args=[First reason why: com.cloudera.cmf.model.DbConfigContainer.configsForDb (#2) has changed]}.
2023-01-25 09:16:52,347 INFO scm-web-107:com.cloudera.cmf.command.flow.CmdStep: Executing command 1546333810 work: Execute 1 steps in sequence
2023-01-25 09:16:52,347 INFO scm-web-107:com.cloudera.cmf.command.flow.CmdStep: Executing command 1546333810 work: Configuration Staleness Check
2023-01-25 09:16:52,347 INFO scm-web-107:com.cloudera.cmf.service.ServiceHandlerRegistry: Global Command ProcessStalenessCheckCommand launched with id=1546333810
2023-01-25 09:16:52,355 INFO CommandPusher-1:com.cloudera.server.cmf.CommandPusherThread: Acquired lease lock on DbCommand:1546333810
2023-01-25 09:16:52,361 INFO ProcessStalenessDetector-0:com.cloudera.cmf.service.config.components.ProcessStalenessDetector: Queuing staleness check with FULL_CHECK for 0/0 roles.
2023-01-25 09:16:52,361 INFO ProcessStalenessDetector-0:com.cloudera.cmf.service.config.components.ProcessStalenessDetector: Staleness check done. Duration: PT0.001S
2023-01-25 09:16:52,361 INFO ProcessStalenessDetector-0:com.cloudera.cmf.service.config.components.ProcessStalenessDetector: Staleness check execution stats: average=0ms, min=0ms, max=0ms.
2023-01-25 09:16:52,365 INFO CommandPusher-1:com.cloudera.server.cmf.CommandPusherThread: Acquired lease lock on DbCommand:1546333810
2023-01-25 09:16:52,365 INFO scm-web-107:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/v45/cm/commands/generateCmca, Status:200
2023-01-25 09:16:52,369 INFO CommandPusher-1:com.cloudera.cmf.model.DbCommand: Command 1546333810(ProcessStalenessCheckCommand) has completed. finalstate:FINISHED, success:true, msg:Successfully finished checking for configuration staleness.
2023-01-25 09:16:52,369 INFO CommandPusher-1:com.cloudera.cmf.command.components.CommandStorage: Invoked delete temp files for command:DbCommand{id=1546333810, name=ProcessStalenessCheckCommand} at dir:/var/lib/cloudera-scm-server/temp/commands/1546333810
2023-01-25 09:17:39,779 INFO pool-6-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Cleaned up
2023-01-25 09:17:40,781 INFO pool-6-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Synced up
2023-01-25 09:18:41,779 INFO pool-6-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Cleaned up
2023-01-25 09:18:42,781 INFO pool-6-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Synced up
2023-01-25 09:19:43,914 INFO pool-6-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Cleaned up
2023-01-25 09:19:44,781 INFO pool-6-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Synced up
2023-01-25 09:20:45,779 INFO pool-6-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Cleaned up
2023-01-25 09:20:46,780 INFO pool-6-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Synced up
2023-01-25 09:21:46,781 INFO pool-6-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (29 skipped) Synced up
2023-01-25 09:21:47,779 INFO pool-6-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Cleaned up
2023-01-25 09:22:48,781 INFO pool-6-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Synced up
2023-01-25 09:22:49,779 INFO pool-6-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Cleaned up
2023-01-25 09:23:50,781 INFO pool-6-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Synced up
2023-01-25 09:23:51,779 INFO pool-6-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Cleaned up
2023-01-25 09:24:43,790 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Reaped total of 0 deleted commands
2023-01-25 09:24:43,804 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Found no commands older than 2021-01-25T08:24:43.790Z to reap.
2023-01-25 09:24:43,804 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Wizard is active, not reaping scanners or configurators
2023-01-25 09:24:52,780 INFO pool-6-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Synced up
2023-01-25 09:24:53,779 INFO pool-6-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Cleaned up
2023-01-25 09:25:54,781 INFO pool-6-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Synced up
2023-01-25 09:25:55,779 INFO pool-6-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Cleaned up
2023-01-25 09:26:54,781 INFO pool-6-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (29 skipped) Synced up
2023-01-25 09:26:57,779 INFO pool-6-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Cleaned up
2023-01-25 09:27:56,780 INFO pool-6-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Synced up
2023-01-25 09:27:59,779 INFO pool-6-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Cleaned up
2023-01-25 09:28:58,780 INFO pool-6-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Synced up
2023-01-25 09:29:01,779 INFO pool-6-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Cleaned up
2023-01-25 09:30:00,780 INFO pool-6-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Synced up
2023-01-25 09:30:03,779 INFO pool-6-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Cleaned up

when logging in CM web UI, I am able to see Add Private Cloud Base Cluster sort of wizard, with:

AutoTLS has already been enabled.
A KDC is currently not configured. This means you cannot create Kerberized clusters.

in /cmf/home there is no cluster added.

when running the same configuration, but using both autotls,tls tags, playbook fails with different error:

TASK [cloudera.cluster.autotls : Enable Auto-TLS] ******************************
Wednesday 25 January 2023  09:21:00 +0000 (0:00:00.217)       0:16:03.749 *****
fatal: [myhost1.domain.com]: FAILED! => {"cache_control": "no-cache, no-store, max-age=0, must-revalidate", "changed": false, "connection": "close", "content": "{\n  \"id\" : 1546333829,\n  \"name\" : \"GenerateCMCACommand\",\n  \"startTime\" : \"2023-01-25T09:21:01.475Z\",\n  \"endTime\" : \"2023-01-25T09:21:16.212Z\",\n  \"active\" : false,\n  \"success\" : false,\n  \"resultMessage\" : \"Failed to enable Auto-TLS\",\n  \"children\" : {\n    \"items\" : [ ]\n  }\n}", "content_type": "application/json;charset=utf-8", "cookies": {"SESSION": "5c861199-d6a7-4084-8ca3-e7fa716d8c08"}, "cookies_string": "SESSION=5c861199-d6a7-4084-8ca3-e7fa716d8c08", "date": "Wed, 25 Jan 2023 09:21:16 GMT", "elapsed": 14, "expires": "Thu, 01 Jan 1970 00:00:00 GMT", "json": {"active": false, "children": {"items": []}, "endTime": "2023-01-25T09:21:16.212Z", "id": 1546333829, "name": "GenerateCMCACommand", "resultMessage": "Failed to enable Auto-TLS", "startTime": "2023-01-25T09:21:01.475Z", "success": false}, "msg": "OK (unknown bytes)", "pragma": "no-cache", "redirected": false, "set_cookie": "SESSION=5c861199-d6a7-4084-8ca3-e7fa716d8c08; Path=/; Secure; HttpOnly", "status": 200, "strict_transport_security": "max-age=31536000 ; includeSubDomains", "url": "https://myhost1.domain.com:7183/api/v45/cm/commands/generateCmca", "x_content_type_options": "nosniff", "x_frame_options": "DENY", "x_xss_protection": "1; mode=block"}

cloudera-scm-server status

● cloudera-scm-server.service - Cloudera CM Server Service
   Loaded: loaded (/usr/lib/systemd/system/cloudera-scm-server.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2023-01-25 10:19:25 CET; 8min ago
  Process: 126100 ExecStartPre=/opt/cloudera/cm/bin/cm-server-pre (code=exited, status=0/SUCCESS)
 Main PID: 126105 (java)
    Tasks: 137
   Memory: 2.5G
   CGroup: /system.slice/cloudera-scm-server.service
           └─126105 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el7_9.x86_64/bin/java -cp .:/usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector...

Jan 25 10:19:25 myhost1 systemd[1]: Starting Cloudera CM Server Service...
Jan 25 10:19:25 myhost1 systemd[1]: Started Cloudera CM Server Service.
Jan 25 10:19:25 myhost1 cm-server[126105]: JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el7_9.x86_64
Jan 25 10:19:25 myhost1 cm-server[126105]: OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
Jan 25 10:19:26 myhost1 cm-server[126105]: ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the co...n logging.
Hint: Some lines were ellipsized, use -l to show in full.

some interesting logs from /var/cloudera-scm-server/cloudera-scm-server.log

2023-01-25 10:20:53,359 INFO scm-web-112:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:PUT, Path:/v45/users/admin
2023-01-25 10:20:53,411 INFO scm-web-112:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:PUT, Path:/v45/users/admin, Status:200
2023-01-25 10:20:56,186 INFO scm-web-115:com.cloudera.server.web.cmf.AuthenticationFailureEventListener: Authentication failure for user: 'admin' from 53.250.49.126
2023-01-25 10:20:58,873 INFO scm-web-128:com.cloudera.server.web.cmf.AuthenticationFailureEventListener: Authentication failure for user: 'admin' from 53.250.49.126
2023-01-25 10:20:59,750 INFO scm-web-104:com.cloudera.server.web.cmf.AuthenticationSuccessEventListener: Authentication success for user: 'admin' from 53.250.49.126
2023-01-25 10:21:00,081 INFO scm-web-105:com.cloudera.server.web.cmf.AuthenticationSuccessEventListener: Authentication success for user: 'admin' from 53.250.49.126
2023-01-25 10:21:01,042 INFO scm-web-110:com.cloudera.server.web.cmf.AuthenticationSuccessEventListener: Authentication success for user: 'admin' from 53.250.49.126
2023-01-25 10:21:01,412 INFO scm-web-119:com.cloudera.server.web.cmf.AuthenticationSuccessEventListener: Authentication success for user: 'admin' from 53.250.49.126
2023-01-25 10:21:01,416 INFO scm-web-119:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/v45/cm/commands/generateCmca
2023-01-25 10:21:01,465 INFO scm-web-119:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing Global command GenerateCMCACommand GenerateCmcaCmdArgs{sshPort=22, userN
ame=root, password=REDACTED, passphrase=REDACTED, privateKey=REDACTED, customCA=false, interpretAsFilenames=true, additionalArguments=null, location=}.
2023-01-25 10:21:01,478 INFO scm-web-119:com.cloudera.cmf.command.flow.CmdStep: Executing command 1546333829 work: Execute 7 steps in sequence
2023-01-25 10:21:01,479 INFO scm-web-119:com.cloudera.cmf.command.flow.CmdStep: Executing command 1546333829 work: Generate a CMCA and enable Auto-TLS.
2023-01-25 10:21:01,487 INFO scm-web-119:com.cloudera.cmf.command.GenerateCmcaCmdWork: Skip disabling init file as host certificate generator was not generate_host_cert
2023-01-25 10:21:01,487 INFO scm-web-119:com.cloudera.cmf.command.GenerateCmcaCmdWork: Storing CMCA in database for HA
2023-01-25 10:21:01,487 INFO scm-web-119:com.cloudera.cmf.command.GenerateCmcaCmdWork: Creating temporary directory for CA generation.
2023-01-25 10:21:01,488 INFO scm-web-119:com.cloudera.cmf.command.GenerateCmcaCmdWork: Generating CMCA
2023-01-25 10:21:01,490 INFO scm-web-119:com.cloudera.cmf.command.CertmanagerRunner: Running CMCA command with args: [setup, --rotate, --configure-services, --skip-cm-init,
 --override, keystore_type=jks]
2023-01-25 10:21:03,076 INFO scm-web-119:com.cloudera.cmf.command.GenerateCmcaCmdWork: Persisting new CMCA to database
2023-01-25 10:21:03,081 INFO scm-web-119:com.cloudera.cmf.command.GenerateCmcaCmdWork: Configuring CM to turn on Auto-TLS
2023-01-25 10:21:03,083 INFO scm-web-119:com.cloudera.cmf.command.GenerateCmcaCmdWork: Setting TLS configuration: AGENT_TLS
2023-01-25 10:21:03,083 INFO scm-web-119:com.cloudera.cmf.command.GenerateCmcaCmdWork: Setting TLS configuration: WEB_TLS
2023-01-25 10:21:03,083 INFO scm-web-119:com.cloudera.cmf.command.GenerateCmcaCmdWork: Setting TLS configuration: NEED_AGENT_VALIDATION
2023-01-25 10:21:03,083 INFO scm-web-119:com.cloudera.cmf.command.GenerateCmcaCmdWork: Setting TLS configuration: KEYSTORE_PATH
2023-01-25 10:21:03,084 INFO scm-web-119:com.cloudera.cmf.command.GenerateCmcaCmdWork: Setting TLS configuration: KEYSTORE_PASSWORD
2023-01-25 10:21:03,084 INFO scm-web-119:com.cloudera.cmf.command.GenerateCmcaCmdWork: Setting TLS configuration: TRUSTSTORE_PATH
2023-01-25 10:21:03,084 INFO scm-web-119:com.cloudera.cmf.command.GenerateCmcaCmdWork: Setting TLS configuration: TRUSTSTORE_PASSWORD
2023-01-25 10:21:03,084 INFO scm-web-119:com.cloudera.cmf.command.GenerateCmcaCmdWork: Setting TLS configuration: HOST_CERT_GENERATOR
2023-01-25 10:21:03,092 INFO scm-web-119:com.cloudera.cmf.command.GenerateCmcaCmdWork: Setting TLS configuration: SSL_CERTIFICATE_HOSTNAME
2023-01-25 10:21:03,096 INFO scm-web-119:com.cloudera.cmf.command.GenerateCmcaCmdWork: Setting TLS configuration: AUTO_TLS_KEYSTORE_PASSWORD
2023-01-25 10:21:03,098 INFO scm-web-119:com.cloudera.cmf.command.GenerateCmcaCmdWork: Setting TLS configuration: AUTO_TLS_TRUSTSTORE_PASSWORD
2023-01-25 10:21:03,101 INFO scm-web-119:com.cloudera.cmf.command.GenerateCmcaCmdWork: Setting TLS configuration: AUTO_TLS_TYPE
2023-01-25 10:21:03,105 INFO scm-web-119:com.cloudera.cmf.command.flow.CmdStep: Executing command 1546333829 work: Generates TLS keys and certificates for a host and instal
l them using SSH
2023-01-25 10:21:03,105 INFO scm-web-119:com.cloudera.cmf.command.GenerateHostCertsCmdWork: Generating host certs for host: myhost1.domain.com
2023-01-25 10:21:03,117 INFO scm-web-119:com.cloudera.cmf.command.GenerateHostCertsCmdWork: Using host certificate generator command: {{TEMP_DIR}}
2023-01-25 10:21:03,117 INFO scm-web-119:com.cloudera.server.cmf.node.HostCertConfigurator: Creating temporary directory for certificate generation.
2023-01-25 10:21:03,126 INFO scm-web-119:com.cloudera.server.cmf.node.HostCertConfigurator: Using host certificate generator command: /opt/cloudera/cm-agent/bin/certmanager
 --location /tmp/generateHostCerts583464968626382515 gen_node_cert --output=-
2023-01-25 10:21:04,451 INFO scm-web-119:net.schmizz.sshj.common.SecurityUtils: BouncyCastle already registered as a JCE provider
2023-01-25 10:21:04,527 INFO scm-web-119:net.schmizz.sshj.transport.TransportImpl: Client identity string: SSH-2.0-SSHJ_0_14_0
2023-01-25 10:21:04,538 INFO scm-web-119:net.schmizz.sshj.transport.TransportImpl: Server identity string: SSH-2.0-OpenSSH_7.4
2023-01-25 10:21:06,975 WARN scm-web-119:com.cloudera.server.cmf.node.SSHConfigurator: Could not authenticate to myhost1.domain.com
net.schmizz.sshj.userauth.UserAuthException: Exhausted available authentication methods

2023-01-25 10:21:06,977 INFO scm-web-119:net.schmizz.sshj.transport.TransportImpl: Disconnected - BY_APPLICATION
2023-01-25 10:21:06,978 WARN scm-web-119:com.cloudera.cmf.command.GenerateHostCertsCmdWork: Error generating certificates. Retrying in 2000 ms.
2023-01-25 10:21:08,979 INFO scm-web-119:net.schmizz.sshj.transport.TransportImpl: Client identity string: SSH-2.0-SSHJ_0_14_0
2023-01-25 10:21:08,996 INFO scm-web-119:net.schmizz.sshj.transport.TransportImpl: Server identity string: SSH-2.0-OpenSSH_7.4
2023-01-25 10:21:10,917 WARN scm-web-119:com.cloudera.server.cmf.node.SSHConfigurator: Could not authenticate to myhost1.domain.com
net.schmizz.sshj.userauth.UserAuthException: Exhausted available authentication methods


2023-01-25 10:21:10,919 INFO scm-web-119:net.schmizz.sshj.transport.TransportImpl: Disconnected - BY_APPLICATION
2023-01-25 10:21:10,920 WARN scm-web-119:com.cloudera.cmf.command.GenerateHostCertsCmdWork: Error generating certificates. Retrying in 3000 ms.
2023-01-25 10:21:13,921 INFO scm-web-119:net.schmizz.sshj.transport.TransportImpl: Client identity string: SSH-2.0-SSHJ_0_14_0
2023-01-25 10:21:13,936 INFO scm-web-119:net.schmizz.sshj.transport.TransportImpl: Server identity string: SSH-2.0-OpenSSH_7.4
2023-01-25 10:21:15,271 INFO LDAP Login Monitor thread:com.cloudera.cmf.service.auth.AbstractExternalServerLoginMonitor: LDAP monitoring is disabled.
2023-01-25 10:21:15,272 INFO KDC Login Monitor thread:com.cloudera.cmf.service.auth.AbstractExternalServerLoginMonitor: KDC monitoring is disabled.
2023-01-25 10:21:16,204 WARN scm-web-119:com.cloudera.server.cmf.node.SSHConfigurator: Could not authenticate to myhost1.domain.com
net.schmizz.sshj.userauth.UserAuthException: Exhausted available authentication methods


2023-01-25 10:21:16,206 INFO scm-web-119:net.schmizz.sshj.transport.TransportImpl: Disconnected - BY_APPLICATION
2023-01-25 10:21:16,212 ERROR scm-web-119:com.cloudera.cmf.command.GenerateHostCertsCmdWork: Error generating certificates: java.lang.IllegalStateException: Not authenticat
ed

2023-01-25 10:21:16,212 ERROR scm-web-119:com.cloudera.cmf.command.flow.WorkOutputs: CMD id: 1546333829 Failed to generate and install host certificates
2023-01-25 10:21:16,212 ERROR scm-web-119:com.cloudera.cmf.model.DbCommand: Command 1546333829(GenerateCMCACommand) has completed. finalstate:FINISHED, success:false, msg:F
ailed to enable Auto-TLS
2023-01-25 10:21:16,218 INFO scm-web-119:com.cloudera.cmf.service.ServiceHandlerRegistry: Global Command GenerateCMCACommand launched with id=1546333829
2023-01-25 10:21:16,241 INFO pool-6-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Cleaned up
2023-01-25 10:21:16,263 INFO scm-web-119:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing Global command ProcessStalenessCheckCommand BasicCmdArgs{args=[First rea
son why: com.cloudera.cmf.model.DbConfig.valueForDb (#1546333786) has changed]}.
2023-01-25 10:21:16,264 INFO scm-web-119:com.cloudera.cmf.command.flow.CmdStep: Executing command 1546333839 work: Execute 1 steps in sequence
2023-01-25 10:21:16,264 INFO scm-web-119:com.cloudera.cmf.command.flow.CmdStep: Executing command 1546333839 work: Configuration Staleness Check
2023-01-25 10:21:16,264 INFO scm-web-119:com.cloudera.cmf.service.ServiceHandlerRegistry: Global Command ProcessStalenessCheckCommand launched with id=1546333839
2023-01-25 10:21:16,275 INFO CommandPusher-1:com.cloudera.server.cmf.CommandPusherThread: Acquired lease lock on DbCommand:1546333839
2023-01-25 10:21:16,281 INFO ProcessStalenessDetector-0:com.cloudera.cmf.service.config.components.ProcessStalenessDetector: Queuing staleness check with FULL_CHECK for 0/0 roles.
2023-01-25 10:21:16,282 INFO ProcessStalenessDetector-0:com.cloudera.cmf.service.config.components.ProcessStalenessDetector: Staleness check done. Duration: PT0.001S
2023-01-25 10:21:16,282 INFO ProcessStalenessDetector-0:com.cloudera.cmf.service.config.components.ProcessStalenessDetector: Staleness check execution stats: average=0ms, min=0ms, max=0ms.
2023-01-25 10:21:16,287 INFO CommandPusher-1:com.cloudera.server.cmf.CommandPusherThread: Acquired lease lock on DbCommand:1546333839
2023-01-25 10:21:16,289 INFO scm-web-119:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/v45/cm/commands/generateCmca, Status:200
2023-01-25 10:21:16,294 INFO CommandPusher-1:com.cloudera.cmf.model.DbCommand: Command 1546333839(ProcessStalenessCheckCommand) has completed. finalstate:FINISHED, success:true, msg:Successfully finished checking for configuration staleness.
2023-01-25 10:21:16,295 INFO CommandPusher-1:com.cloudera.cmf.command.components.CommandStorage: Invoked delete temp files for command:DbCommand{id=1546333839, name=ProcessStalenessCheckCommand} at dir:/var/lib/cloudera-scm-server/temp/commands/1546333839
2023-01-25 10:21:17,244 INFO pool-6-thread-1:com.cloudera.server.cmf.components.CmServerStateSynchronizer: (30 skipped) Synced up
2023-01-25 10:21:50,642 INFO avro-servlet-hb-processor-3:com.cloudera.server.common.AgentAvroServlet: (25 skipped) AgentAvroServlet: heartbeat processing stats: average=21ms, min=4ms, max=155ms.

2023-01-25 10:57:24,606 ERROR ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: Unable to retrieve remote parcel repository manifest
2023-01-25 10:59:22,544 ERROR main:com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean: Could not read license file /etc/cloudera-scm-server/license.txt
2023-01-25 11:00:10,064 ERROR ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: Unable to retrieve remote parcel repository manifest
2023-01-25 11:00:12,183 WARN MainThread:org.eclipse.jetty.security.SecurityHandler: ServletContext@o.e.j.s.ServletContextHandler@fa85d63{/,null,STARTING} has uncovered http methods for path: /*
2023-01-25 11:00:12,399 ERROR MainThread:com.cloudera.enterprise.TLSUtil: Could not determine if current JDK can perform secure SSL/TLS renegotiation. Defaulting to no-renegotiations.
2023-01-25 11:00:12,514 WARN WebServerImpl:org.eclipse.jetty.security.SecurityHandler: ServletContext@o.e.j.w.WebAppContext@2c9d79fb{/,file:///opt/cloudera/cm/webapp/,STARTING}{/opt/cloudera/cm/webapp} has uncovered http methods for path: /*

I have also tried putting whole private key file content into variable host_ssh_private_key created in

/opt/cldr-runner/collections/ansible_collections/cloudera/cluster/roles/cloudera_manager/autotls/defaults/main.yml

https://github.com/cloudera-labs/cloudera.cluster/blob/main/roles/cloudera_manager/autotls/defaults/main.yml

and used this variable in this file

/opt/cldr-runner/collections/ansible_collections/cloudera/cluster/roles/cloudera_manager/autotls/templates/request.j2

https://github.com/cloudera-labs/cloudera.cluster/blob/main/roles/cloudera_manager/autotls/templates/request.j2

Private key content had to be as one-line with \\n instead of newlines.

when running with tags default_cluster,kerberos,autotls,tls, with tls=true in inventory_static.ini and tls: true
in security section of cluster/mgmt cluster definitions. got the following error:

TASK [cloudera.cluster.autotls : Enable Auto-TLS] ******************************
Friday 27 January 2023  14:40:10 +0000 (0:00:00.095)       0:15:03.414 ********
fatal: [myhost1.domain.com]: FAILED! => {"cache_control": "no-cache, no-store, max-age=0, must-revalidate", "changed": false, "connection": "close", "content": "{\n  \"id\" : 1546333829,\n  \"name\" : \"GenerateCMCACommand\",\n  \"startTime\" : \"2023-01-27T14:40:10.982Z\",\n  \"endTime\" : \"2023-01-27T14:40:19.660Z\",\n  \"active\" : false,\n  \"success\" : false,\n  \"resultMessage\" : \"Failed to enable Auto-TLS\",\n  \"children\" : {\n    \"items\" : [ ]\n  }\n}", "content_type": "application/json;charset=utf-8", "cookies": {"SESSION": "698ea13f-400c-4f63-aa9c-b69f6efd2cf4"}, "cookies_string": "SESSION=698ea13f-400c-4f63-aa9c-b69f6efd2cf4", "date": "Fri, 27 Jan 2023 14:40:19 GMT", "elapsed": 8, "expires": "Thu, 01 Jan 1970 00:00:00 GMT", "json": {"active": false, "children": {"items": []}, "endTime": "2023-01-27T14:40:19.660Z", "id": 1546333829, "name": "GenerateCMCACommand", "resultMessage": "Failed to enable Auto-TLS", "startTime": "2023-01-27T14:40:10.982Z", "success": false}, "msg": "OK (unknown bytes)", "pragma": "no-cache", "redirected": false, "set_cookie": "SESSION=698ea13f-400c-4f63-aa9c-b69f6efd2cf4; Path=/; Secure; HttpOnly", "status": 200, "strict_transport_security": "max-age=31536000 ; includeSubDomains", "url": "https://myhost1.domain.com:7183/api/v45/cm/commands/generateCmca", "x_content_type_options": "nosniff", "x_frame_options": "DENY", "x_xss_protection": "1; mode=block"}

logs:

2023-01-27 15:40:19,652 WARN scm-web-114:com.cloudera.server.cmf.node.SSHConfigurator: Could not authenticate to myhost1.domain.com
net.schmizz.sshj.userauth.UserAuthException: Exhausted available authentication methods


Caused by: net.schmizz.sshj.userauth.UserAuthException: Problem getting public key from PKCS8KeyFile{resource=[PrivateKeyStringResource]}


Caused by: java.io.IOException: unrecognised object: OPENSSH PRIVATE KEY



2023-01-27 15:40:19,654 INFO scm-web-114:net.schmizz.sshj.transport.TransportImpl: Disconnected - BY_APPLICATION
2023-01-27 15:40:19,660 ERROR scm-web-114:com.cloudera.cmf.command.GenerateHostCertsCmdWork: Error generating certificates: java.lang.IllegalStateException: Not authenticat
ed

2023-01-27 15:40:19,660 ERROR scm-web-114:com.cloudera.cmf.command.flow.WorkOutputs: CMD id: 1546333829 Failed to generate and install host certificates
2023-01-27 15:40:19,660 ERROR scm-web-114:com.cloudera.cmf.model.DbCommand: Command 1546333829(GenerateCMCACommand) has completed. finalstate:FINISHED, success:false, msg:Failed to enable Auto-TLS

Caused by: java.io.IOException: unrecognised object: OPENSSH PRIVATE KEY indicates that CM somehow still cannot read the private key.

hadoopch · 2023-09-13T12:35:56Z

hadoopch
Sep 13, 2023

Hello,

it is correct to set tls to false if you want to use autotls.

But there are some issues.

For autotls you need to create a user with appropriate sudo rights and a password.
Furthermore you have to define the following vars :

host_ssh_username: "{{cdpcfg.install_user}}"
host_ssh_password: "{{sec_cdpcfg['install_user']}}"

The request.j2 Template does not contain variables for private key resp. passphrase of the private key.

request.j2

So you have to work with password or adjust the template

Furthermore you have to set:

become: true

Otherwise the cloudera manager server does not restart.

Additionally the order in cluster.yml playbook is not good for autotls.
The autotls role includes a restart of CMS.
If you run autotls at a early stage of the playbook then CMS is still missing and the play will fail

See

Finally after the implementation of autotls a restart is required and client configuration has to be redeployed

To get it running (quick and dirty) i put the following lines into my application.yml

- name: Enable Auto-TLS
  hosts: cloudera_manager
  gather_facts: no
  become: true
  roles:
    - cloudera.cluster.cloudera_manager.autotls
  tags:
    - ATLS
    - never

- name: Restart and re-deploy stale client configs. After enabling autotls
  hosts: localhost
  gather_facts: no
  roles:
    - role: cloudera.cluster.operations.restart_stale
  tags:
    - ATLS
    - never
    - RESTART

After the Cluster is complete and i am able to connect to CM via HTTP i run the playbook again with:

--tags ATLS

and autols will be configured and services restarted

Hope this helps

Regards

Uli

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot deploy with AutoTLS #141

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Cannot deploy with AutoTLS #141

HirossxD Jan 30, 2023

Replies: 1 comment

hadoopch Sep 13, 2023

HirossxD
Jan 30, 2023

hadoopch
Sep 13, 2023