Draft: Move to mongosh and support mongodb versions 4.4+ only

[witjoh]

This is a breaking PR

• move to mongodb-mongosh (mongo will not work anymore)
• Only support mongodb version 4.4 and higher (following mongodb support cycle)
• Still work in process
• Tested on rhel8 an mongodb 6.x

[tsgoff]

Tested with Ubuntu 20.04 aarch64 and 6.0.8
Install, database and user creation is working.
Replication configuration is failing:
Error: /Stage[main]/Mongodb::Replset/Mongodb_replset[sre-mongodb.int]: Could not evaluate: Can't connect to any member of replicaset sre-mongodb.int.

[witjoh]

@tsgoff Thanks for testing.
I assume the replicset is already initiated at that time, otherwise no user could be created.
What does the rs.status()/rs.conf() executed from mongosh returns ?
Does running puppet with --debug option gives more output ?

[tsgoff]

@tsgoff Thanks for testing. I assume the replicset is already initiated at that time, otherwise no user could be created. What does the rs.status()/rs.conf() executed from mongosh returns ? Does running puppet with --debug option gives more output ?

test> rs.status()
MongoServerError: not running with --replSet

Debug: Lookup of 'mongodb::hosts'
  Searching for "lookup_options"
    Global Data Provider (hiera configuration version 5)
      Using configuration "/etc/puppet/hiera.yaml"
      Hierarchy entry "app"
        Path "/etc/puppet/hiera/app.yaml"
          Original path: "app.yaml"
          No such key: "lookup_options"
    Module data provider for module "mongodb" not found
  Searching for "mongodb::hosts"
    Global Data Provider (hiera configuration version 5)
      Using configuration "/etc/puppet/hiera.yaml"
      Hierarchy entry "app"
        Path "/etc/puppet/hiera/app.yaml"
          Original path: "app.yaml"
          Found key: "mongodb::hosts" value: [
            {
              "host" => "172.30.1.106:27017"
            }
          ]

Debug: Prefetching mongo resources for mongodb_replset
Debug: Executing: '/usr/bin/mongosh admin --quiet --host demo-mongodb-01-int.example.com:27017 --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.conf())'
Debug: Request failed: 'Execution of '/usr/bin/mongosh admin --quiet --host demo-mongodb-01-int.example.com:27017 --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.conf())' returned 1: MongoServerError: no replset config has been received' Retry: '1'
Debug: Executing: '/usr/bin/mongosh admin --quiet --host demo-mongodb-01-int.example.com:27017 --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.conf())'
Debug: Request failed: 'Execution of '/usr/bin/mongosh admin --quiet --host demo-mongodb-01-int.example.com:27017 --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.conf())' returned 1: MongoServerError: no replset config has been received' Retry: '2'
Debug: Executing: '/usr/bin/mongosh admin --quiet --host demo-mongodb-01-int.example.com:27017 --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.conf())'
Debug: Request failed: 'Execution of '/usr/bin/mongosh admin --quiet --host demo-mongodb-01-int.example.com:27017 --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.conf())' returned 1: MongoServerError: no replset config has been received' Retry: '3'
Debug: Executing: '/usr/bin/mongosh admin --quiet --host demo-mongodb-01-int.example.com:27017 --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.conf())'
Notice: /Stage[main]/Mongodb::Replset/Mongodb_replset[mongodb.int]/ensure: created
Debug: Checking for dead and alive members
Debug: Checking replicaset member 172.30.1.106:27017 ...
Debug: Executing: '/usr/bin/mongosh admin --quiet --host 172.30.1.106:27017 --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.status())'
Debug: Request failed: 'Execution of '/usr/bin/mongosh admin --quiet --host 172.30.1.106:27017 --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.status())' returned 1: MongoServerError: no replset config has been received' Retry: '1'
Debug: Executing: '/usr/bin/mongosh admin --quiet --host 172.30.1.106:27017 --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.status())'
Debug: Request failed: 'Execution of '/usr/bin/mongosh admin --quiet --host 172.30.1.106:27017 --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.status())' returned 1: MongoServerError: no replset config has been received' Retry: '2'
Debug: Executing: '/usr/bin/mongosh admin --quiet --host 172.30.1.106:27017 --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.status())'
Debug: Request failed: 'Execution of '/usr/bin/mongosh admin --quiet --host 172.30.1.106:27017 --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.status())' returned 1: MongoServerError: no replset config has been received' Retry: '3'
Debug: Executing: '/usr/bin/mongosh admin --quiet --host 172.30.1.106:27017 --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.status())'
Debug: Alive members: []
Debug: Dead members: [{"host"=>"172.30.1.106:27017"}]
Error: /Stage[main]/Mongodb::Replset/Mongodb_replset[mongodb.int]: Could not evaluate: Can't connect to any member of replicaset mongodb.int.
Debug: Class[Mongodb::Replset]: Resource is being skipped, unscheduling all events
Debug: Mongodb::Db[admin]: Resource is being skipped, unscheduling all events
Debug: Mongodb::Db[admin]: Resource is being skipped, unscheduling all events
Debug: Class[Mongodb::Server]: Resource is being skipped, unscheduling all events
Notice: /Stage[main]/Ec2mongodb/File[/var/log/mongodb/mongod.log]: Dependency Mongodb_replset[mongodb.int] has failures: true

Puppet part:

  class { '::mongodb::globals':
    manage_package_repo => true,
    version             => '6.0.0',
  } ->
  class { '::mongodb::client': } ->
  class { '::mongodb::server':
    ensure         => present,
    restart        => false,
    auth           => true,
    replset        => "${replicaset}",
    replset_config => {
      "${replicaset}" => {
        ensure  => present,
        members => $mongo_array
      }
    },
    create_admin   => true,
    admin_username => 'admin',
    admin_password => "${adminpw}",
    admin_roles    => ['root'],
    dbpath         => $dbpath,
    storage_engine => 'wiredTiger',
    bind_ip        => ['0.0.0.0'],
    store_creds    => true,
    keyfile        => '/etc/mongo-key',
    key            => "${cluster_key}",
  }

Generated /etc/mongod.conf

#mongodb.conf - generated from Puppet
#System Log
systemLog.path: /var/log/mongodb/mongodb.log
systemLog.destination: file
systemLog.logAppend: true
systemLog.quiet: true
#Process Management
processManagement:
  pidFilePath: /var/run/mongod.pid
#Storage
storage.dbPath: /data/db/mongodb
storage.engine: wiredTiger
#Security
security.authorization: enabled
security.keyFile: /etc/mongo-key
#Net
net.bindIp:  0.0.0.0
#Replication
replication.replSetName: mongodb.int
#Sharding
#Operation Profiling

[witjoh]

Debug: Executing: '/usr/bin/mongosh admin --quiet --host 172.30.1.106:27017 --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.status())'
Debug: Alive members: []
Debug: Dead members: [{"host"=>"172.30.1.106:27017"}]

Replicaset is not initiated. Since auth is enabled, but the admin user is not yet created (we need the replicaset initiated first to be able to do that), it ss only possible to connect to mongod over localhost. But we still try to connect over the IP connection. Rescue is working since we can only use rs.status() at this point.

need to check the code why its not connecting over localhost.

[tsgoff]

Replicaset is not initiated. Since auth is enabled, but the admin user is not yet created (we need the replicaset initiated first to be able to do that), it ss only possible to connect to mongod over localhost. But we still try to connect over the IP connection. Rescue is working since we can only use rs.status() at this point.

with older versions it worked to stop the daemon, create the admin user with mongo shell, set a lock file and restart mongodb with another puppet apply.

old version without mongosh: https://raw.githubusercontent.com/tsgoff/mongodb-demo/master/.deployment/codedeploy/init.sh

In my test case the admin user is created and mongosh can authenticate with the .mongosh.rc

[witjoh]

with older versions it worked to stop the daemon, create the admin user with mongo shell, set a lock file and restart mongodb with another puppet apply.

Current method is based on : https://www.mongodb.com/docs/v6.0/core/localhost-exception/#std-label-localhost-exception

It is I think still possible to use the method you described, but that not sure if that shoudl be implemented in the providers. It works on Redhat with current code, so I'm a but surprised it doesn't on ubuntu.

Still struggling to setup local aceptance test environment, so no ubuntu/debian available yet for testing the code properly.

[tsgoff]

With auth -> false I get

Error: Failed to apply catalog: Could not evaluate MongoDB shell command: load('/root/.mongoshrc.js'); EJSON.stringify(rs.status()) with Execution of '/usr/bin/mongosh admin --quiet --host demo-mongodb-01-int.example.com:27017 --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.status())' returned 1: MongoServerError: command replSetGetStatus requires authentication

root@demo-mongodb-01-int:/data/deploy# mongosh admin --quiet --host demo-mongodb-01-int.example.com:27017 --eval "load('/root/.mongoshrc.js'); EJSON.stringify(rs.status())"
MongoServerError: command replSetGetStatus requires authentication
root@demo-mongodb-01-int:/data/deploy# mongosh admin --quiet --host 127.0.0.1:27017 --eval "load('/root/.mongoshrc.js'); EJSON.stringify(rs.status())"
MongoServerError: no replset config has been received
root@demo-mongodb-01-int:/data/deploy# mongosh admin --quiet --host 172.30.1.104:27017 --eval "load('/root/.mongoshrc.js'); EJSON.stringify(rs.status())"
MongoServerError: command replSetGetStatus requires authentication

/root/.mongoshrc.js

function rsReconfigMember(member){
  var cfg = rs.config()
  cfg.members.forEach(function(part,index,memberArray){
    if (member.host == part.host) {
      for(k in member){
        memberArray[index][k] = member[k]
      }
    }
  })
  return rs.reconfig(cfg)
}

function rsReconfigSettings(settings){
  var cfg = rs.config()
  cfg.settings = settings
  return rs.reconfig(cfg)
}

Mongosh command on disabled auth is only working without host, 127.0.0.1, localhost but not with actual hostname or host ip.

[flepoutre]

Hello,
I have tested it on Ubuntu 22.04, the installation works but I have some errors when I try to create databases and users, for example :

Error: /Stage[main]/Mongodb::Server/Mongodb::Db[admin]/Mongodb_database[admin]: Could not evaluate: No ability to determine if mongodb_database exists


Do you have any idea about this ?

Thanks.

[Wimmesberger]

I've tested your branch for an existing 4.4 cluster using TLS with BasicAuth and later upgraded it to 5.0 and 6.0 on Ubuntu 20.04.
I had to apply 2 patches to make it work:

1. $config is not defined, I think you meant to use config in lib/puppet/provider/mongodb.rb:114

diff --git a/lib/puppet/provider/mongodb.rb b/lib/puppet/provider/mongodb.rb
index ec0ed3f..9cae1ea 100644
--- a/lib/puppet/provider/mongodb.rb
+++ b/lib/puppet/provider/mongodb.rb
@@ -111,7 +111,7 @@ class Puppet::Provider::Mongodb < Puppet::Provider
       args.push('--tlsAllowInvalidHostnames') if tls_invalid_hostnames(config)
     end

-    if $config['auth_mechanism'] && $config['auth_mechanism'] == 'x509'
+    if config['auth_mechanism'] && config['auth_mechanism'] == 'x509'
       args.push("--authenticationDatabase '$external' --authenticationMechanism MONGODB-X509")
     end

2. mongosh with tls behaves differently compared the old mongo client as it defaults to an empty truststore and needs either --tlsCAFile, --tlsUseSystemCA or --tlsAllowInvalidCertificates in order to succesfully connect. The module does add --tlsCAFile but only if net.tls.ca_file is set in the mongod.conf however that also enables client cert validation which I don't want.
   To solve this for my usecase I've added the --tlsUseSystemCA flag if net.tls.CAFile is nil, however this fix might not be good enough in case the custom CA is not contained in the system truststore. It would probably be better if you'd be able to configure setting --tlsCAFile specific to the client needs.

diff --git a/lib/facter/is_master.rb b/lib/facter/is_master.rb
index a42c04d..c251710 100644
--- a/lib/facter/is_master.rb
+++ b/lib/facter/is_master.rb
@@ -42,6 +42,7 @@ def get_options_from_hash_config(config)
   result << "--tls --host #{Facter.value(:fqdn)}" if config['net.tls.mode'] == 'requireTLS' || !tlscert.nil? || !config['net.tls.CAFile'].nil?
   result << "--tlsCertificateKeyFile #{tlscert}" unless tlscert.nil?
   result << "--tlsCAFile #{config['net.tls.CAFile']}" unless config['net.tls.CAFile'].nil?
+  result << "--tlsUseSystemCA" if config['net.tls.CAFile'].nil?

   # use --authenticationMechanism, ---authenticationDatabase
   # when
diff --git a/lib/puppet/provider/mongodb.rb b/lib/puppet/provider/mongodb.rb
index 9cae1ea..2bd5d9d 100644
--- a/lib/puppet/provider/mongodb.rb
+++ b/lib/puppet/provider/mongodb.rb
@@ -106,6 +106,7 @@ class Puppet::Provider::Mongodb < Puppet::Provider

       tls_ca = config['tlsca']
       args += ['--tlsCAFile', tls_ca] unless tls_ca.nil?
+      args += ['--tlsUseSystemCA'] if tls_ca.nil?
       args += ['--tlsCertificateKeyFile', config['tlscert']]

       args.push('--tlsAllowInvalidHostnames') if tls_invalid_hostnames(config)

bonus, this one is cosmetic: you've added an additional empty line in templates/mongodb.conf.erb:179 which leads to a mongod restart (the refresh/restart does happen anyway due how the relation to the client is defined)

[witjoh]

@Wimmesberger Thank you ...
I will have a look at this more deeply.
It is indeed better to not force client cert validation.
If I remember well, multiple authenticationMechanism can be set in the /etc/mongod.conf ... Think I already looked into that. Have to check if this can also be used ....
Will get back in touch ...

[h-haaks]

I was just browsing PRs in this module trying to figure out why it doesn't have stdlib 9 support yet.
Then I stumbled into this one :)
It took me some time to read it trough and then it struck me..

Wouldn't it be better to implement a second set of providers targeting mongosh instead of changing existing ones?
That way we could do this in smaller steps and the users of the module have a way to select implementation in the beginning.

[whiphubley]

Please note with MongoDB 7x you need to remove the storage.journal.enabled config option as it's entirely removed and the server won't actually start...

diff --git a/modules/mongodb/manifests/server.pp b/modules/mongodb/manifests/server.pp
index fb11ee2b..4d4335d4 100644
--- a/modules/mongodb/manifests/server.pp
+++ b/modules/mongodb/manifests/server.pp
@@ -379,2 +379,3 @@ class mongodb::server (
   Optional[Boolean] $nojournal                                       = undef,
+  Optional[Boolean] $removejournal                                   = undef,
   Optional[Boolean] $smallfiles                                      = undef,
diff --git a/modules/mongodb/manifests/server/config.pp b/modules/mongodb/manifests/server/config.pp
index 18e830bc..d7ed2420 100644
--- a/modules/mongodb/manifests/server/config.pp
+++ b/modules/mongodb/manifests/server/config.pp
@@ -21,2 +21,3 @@ class mongodb::server::config {
   $nojournal             = $mongodb::server::nojournal
+  $removejournal         = $mongodb::server::removejournal
   $smallfiles            = $mongodb::server::smallfiles
diff --git a/modules/mongodb/templates/mongodb.conf.erb b/modules/mongodb/templates/mongodb.conf.erb
index 709e688b..320d233e 100644
--- a/modules/mongodb/templates/mongodb.conf.erb
+++ b/modules/mongodb/templates/mongodb.conf.erb
@@ -48,2 +48,5 @@ storage.dbPath: <%= @dbpath %>
 storage.journal.enabled: false
+<% elsif @removejournal -%>
+#Removed in 6x as Mongo now always enables journaling.
+#storage.journal.enabled: true
 <% elsif @journal -%>

I comment out the directive for clarity but you'll probably just wish to remove it. If you don't you get this when attempting to start the server...

Unrecognized option: storage.journal.enabled

[whiphubley]

Also if you choose not to use client certs you don't want to pass the --tlsCertificateKeyFile directive when connecting ( it just hangs and finally times out ) so I had to force this with the following...

diff --git a/modules/mongodb7plus/lib/puppet/provider/mongodb.rb b/modules/mongodb7plus/lib/puppet/provider/mongodb.rb
index 7448ca20..84918c7f 100644
--- a/modules/mongodb7plus/lib/puppet/provider/mongodb.rb
+++ b/modules/mongodb7plus/lib/puppet/provider/mongodb.rb
@@ -107,7 +107,7 @@ class Puppet::Provider::Mongodb < Puppet::Provider
       tls_ca = config['tlsca']
       args += ['--tlsCAFile', tls_ca] unless tls_ca.nil?
       args += ['--tlsUseSystemCA'] if tls_ca.nil?
-      args += ['--tlsCertificateKeyFile', config['tlscert']]
+      #args += ['--tlsCertificateKeyFile', config['tlscert']]

^^ You probably want to check for allowConnectionsWithoutCertificates in the config and then not apply the above if true.

[whiphubley]

Another minor issue...if you want the tlsUseSystemCA param to be added to the config then it needs to be available to the erb template via the config file...

diff --git a/modules/mongodb/manifests/server/config.pp b/modules/mongodb/manifests/server/config.pp
index d7ed2420..b4ab03cc 100644
--- a/modules/mongodb/manifests/server/config.pp
+++ b/modules/mongodb/manifests/server/config.pp
@@ -77,4 +77,5 @@ class mongodb::server::config {
   $tls_invalid_hostnames = $mongodb::server::tls_invalid_hostnames
   $tls_mode              = $mongodb::server::tls_mode
+  $tls_use_system_ca     = $mongodb::server::tls_use_system_ca
   $storage_engine        = $mongodb::server::storage_engine

[h-haaks]

@witjoh could you move 07eac43 into a separate PR?
Would really like to include that in the next release.

[whiphubley]

Can we please resolve the following...

1. When auth is enabled the config and .rc files are updated to reflect this.
2. The module then attempts to create the admin db and user.
3. It can't do this because auth is enabled.

Can we not create the necessary admin credentials before the config file is updated with auth: true ?

This is a chicken and egg situation which constantly causes problems.

The only solution is to disable auth / run the module / create the databases / enable auth.

[witjoh]

Can we please resolve the following...

1. When auth is enabled the config and .rc files are updated to reflect this.

2. The module then attempts to create the admin db and user.

3. It can't do this because auth is enabled.

Can we not create the necessary admin credentials before the config file is updated with auth: true ?

This is a chicken and egg situation which constantly causes problems.

The only solution is to disable auth / run the module / create the databases / enable auth.

This should be fixed in #703 which should be merged soon ...

[whiphubley]

Can we please resolve the following...

1. When auth is enabled the config and .rc files are updated to reflect this.

2. The module then attempts to create the admin db and user.

3. It can't do this because auth is enabled.

Can we not create the necessary admin credentials before the config file is updated with auth: true ?
This is a chicken and egg situation which constantly causes problems.
The only solution is to disable auth / run the module / create the databases / enable auth.

This should be fixed in #703 which should be merged soon ...

OK will check and test again but I merged the entire #677 PR so pretty sure I have it. Either way I'll provide a more detailed report after the next run thanks.

[stevenpost]

@h-haaks

@witjoh could you move 07eac43 into a separate PR? Would really like to include that in the next release.

See #709

[whiphubley]

Can we please resolve the following...

1. When auth is enabled the config and .rc files are updated to reflect this.

2. The module then attempts to create the admin db and user.

3. It can't do this because auth is enabled.

Can we not create the necessary admin credentials before the config file is updated with auth: true ?
This is a chicken and egg situation which constantly causes problems.
The only solution is to disable auth / run the module / create the databases / enable auth.

This should be fixed in #703 which should be merged soon ...

@witjoh so I can see your new check in mongodb.rb for "requires_authentication" message in mongodb.rb....and this is itself in an IF statement for the is_master fact....but I never evaluate it due to the following...

# facter -p mongodb_is_master
not_installed

And checking the code you are running a Facter::Util::Resolution for mongosh and mongod...but I have those...

# which mongosh
/usr/bin/mongosh

# which mongod
/usr/bin/mongod

Any ideas why it's not "resolving" those binaries ?

[stevenpost]

@whiphubley I took over work on this from @witjoh.
Please see #703 for my work in progress. I stripped out a lot of unrelated stuff, that will go in separate PRs, some cleanups are already merged.

Also, thank you for testing and your insights, I learned a lot from your comments. Once the migration to mongosh is complete, I'll add in proper support for MongoDB 6 and maybe 7.

[witjoh]

handled by #703. this can be closed now since mentioned PR is eady to be merged.