Unable to collect data from MongoDB Arbiter with auth enabled

[pnedkov]

Relevant telegraf.conf:

System info:

Telegraf 1.6.2
CentOS 7.5.1804
Three MongoDB 3.2.20 nodes configured with "auth=true" and "keyFile=/var/lib/mongo-keyfile" - primary, secondary and arbiter

Steps to reproduce:

1. Configure MongoDB with 3 nodes in a replica set (Primary, Secondary and Arbiter) with the following options:

replSet=rs0
auth=true
keyFile=/var/lib/mongo-keyfile

2. Try to monitor the Arbiter with the following Telegraf configuration:

[[inputs.procstat]]
    exe = "mongod"

[[inputs.mongodb]]
    servers = ["mongodb://127.0.0.1:27017"]

Expected behavior:

No errors in telegraf.log.

Actual behavior:

E! Error getting database shard stats (not authorized on admin to execute command { shardConnPoolStats: 1 })

Additional info:

I have "u1" user in "prod" db. On Mongo Primary and Secondary nodes I can successfully run Telegraf with no errors with the following configuration:

[[inputs.procstat]]
    exe = "mongod"

[[inputs.mongodb]]
    servers = ["mongodb://u1:u1password@127.0.0.1:27017/prod"]

But the Arbiter has no "prod" db and neither it has "admin" db.

How am I supposed to do this on the Arbiter?
I am sorry if this is not a bug but just a user misconfiguration instead.

[danielnelson]

I do have this working on an arbiter, though it doesn't have the storage fully turned off as it was just a regular replica that was added as an arbiter.

> mongodb,hostname=debian-stretch-mongodb-3:27017 wtcache_pages_evicted_by_app_thread=0i,active_writes=0i,net_in_bytes=296i,updates_per_sec=0i,repl_deletes_per_sec=0i,wtcache_worker_thread_evictingpages=0i,total_in_use=0i,wtcache_tracked_dirty_bytes=0i,wtcache_bytes_written_from=205186i,getmores_per_sec=0i,repl_getmores_per_sec=0i,repl_lag=0i,repl_queries_per_sec=0i,repl_inserts_per_sec=0i,net_out_bytes=22707i,wtcache_pages_queued_for_eviction=0i,wtcache_app_threads_page_read_time=1594i,ttl_deletes_per_sec=0i,queued_reads=0i,active_reads=0i,inserts_per_sec=0i,flushes_per_sec=0i,wtcache_bytes_read_into=69398i,wtcache_current_bytes=149886i,vsize_megabytes=720i,open_connections=3i,repl_commands_per_sec=0i,percent_cache_used=0,wtcache_server_evicting_pages=0i,total_available=0i,total_created=0i,commands_per_sec=3i,deletes_per_sec=0i,state="ARBITER",jumbo_chunks=0i,wtcache_app_threads_page_read_count=14i,wtcache_max_bytes_configured=1073741824i,wtcache_app_threads_page_write_count=0i,ttl_passes_per_sec=0i,queued_writes=0i,repl_updates_per_sec=0i,member_status="ARB",total_refreshing=0i,resident_megabytes=62i,queries_per_sec=0i,percent_cache_dirty=0 1526689805000000000

Can you add a user with the appropriate permissions to the admin database on the arbiter?:

$ mongo admin -u <user> -p
> use admin
> db.createUser({user: "u1", pwd: "u1password", roles: [{role:"clusterMonitor",db:"admin"}]})

[pnedkov]

I am afraid the creation of users on Arbiter is not allowed. From here:

Because arbiters do not store data, they do not possess the internal table of user and role mappings used for authentication. Thus, the only way to log on to an arbiter with authorization active is to use the localhost exception.

[danielnelson]

Thanks for the update. I guess you will either need to turn off authentication or you could try doing what I did: setup a real replica set member and then convert it to and arbiter, which seems to keep the users intact.

[pnedkov]

I think I found a solution. I will just create the user on the "arbiter" while the server is still running in a stand-alone mode and not part of a replica set. Once the arbiter is configured I can login with the user I have created and telegraf no longer complains.

[pnedkov]

By the way, the problem is only reproducible with Telegraf 1.6.x. On another server where Telegraf is 1.5.2 there are no issues. I just upgraded to Telegraf 1.6.3 on that server and BOOM:

2018-05-21T21:10:10Z E! Error getting database shard stats (not authorized on admin to execute command { shardConnPoolStats: 1 })

Back to 1.5.2 and no issues again:

yum downgrade -y telegraf-1.5.2
systemctl restart telegraf.service

[danielnelson]

That makes sense, we only added these shard stats in 1.6. Due to how hard it is to setup the user in this case perhaps we should ignore the error.

[pnedkov]

With Telegraf 1.8.0 I get one additional error message:

2018-09-24T18:00:10Z E! Error getting first oplog entry (not authorized on local to execute command { find: "oplog.rs", filter: { ts: { $exists: true } }, sort: { $natural: 1 }, skip: 0, limit: 1, batchSize: 1, singleBatch: true })
2018-09-24T18:00:20Z E! Error getting database shard stats (not authorized on admin to execute command { shardConnPoolStats: 1 })

Is there an option I could use to either suppress these error messages in the log file or even better to disable these data collections that are not relevant for the arbiter? It would have been nice if the mongo plugin itself does not try to collect this data if it detects that the Mongo node is an arbiter.

Removing all our Mongo Arbiters from their replica set only to create a user to make Telegraf happy seems excessive to me. Telegraf collects all the data we need from our Mongo Arbiters even with the error messages every 10 seconds.

[danielnelson]

Sorry, there is no option to disable the logging, we will get it fixed.