Default soft limit for open files is 1024

[plars]

Steps to reproduce

1. connect to the running mongod server using mongosh (credentials may be needed, get these from juju unit data, etc)
2. Message pops up warning:

        2023-08-28T05:11:08.183+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem
        2023-08-28T05:11:12.730+00:00: Soft rlimits for open file descriptors too low
        2023-08-28T05:11:12.730+00:00:         currentValue: 1024
        2023-08-28T05:11:12.730+00:00:         recommendedMinimum: 64000

Adjust /etc/security/limits.conf to increase this to 64000. I just used "* soft nofile 64000" for now, but it would probably be better to restrict this further. This resulted in any other processes showing 64000 as the new soft limit, but when I restart the mongod service or reboot the instance, it doesn't increase this for mongod running in the snap. You can also confirm this by getting the pid of the mongod process and running cat /proc/<pid>/limits

Expected behavior

MongoDB recommends a minimum of 64000 for the NOFILES soft limit (num open files). See http://dochub.mongodb.org/core/prodnotes-filesystem
This might need further configuration - but certainly keeping it as low as 1024 seems to have known issues with causing crashes for many people. A more appropriate default for the charm should be used.

Actual behavior

Default NOFILES seems to be 1024

Versions

5/edge (rev 126)

Operating system:
20.04.6

Juju CLI: 2.9.44-ubuntu-amd64

Juju agent: 2.9.44

Charm revision: 126

LXD: 4.0.9

Log output

Juju debug log:
(not relevant here)

Additional context

Workaround is possible by editing /etc/systemd/system/snap.charmed-mongodb.mongod.service and adding LimitNOFILE=64000 in the Service section. Then run systemctl daemon-reload and restart the service. After doing this, I can confirm that the open files limit is increased and the error goes away.

[github-actions]

https://warthogs.atlassian.net/browse/DPE-2466

[plars]

Just noticed that the workaround I mentioned doesn't seem to be durable. I modified the service file a while back and it seems to have been reset back to default at some point, possibly though a charm update. I haven't tried using an override yet, but it's still just a workaround. Based on this being a recommended setting from mongodb, I wonder if it should just be the default in the charmd-mongodb snap