How to clean up database properly

[mman]

I know this might be more of a question then actual bug report, and I know there are other tickets open for this, but let me open it anyway, feel free to close or suggest workarounds. I'm trying to better understand the inner workings of Parse Server to come up with appropriate PRs to help make the Parse Server run smoothly over time without need for extra maintenance. I am sure many of you have come up with their own home grown solutions.

Problem Summary

Parse Server Database is growing over time and needs to be cleaned up manually. Typically via a cloud code function that is being invoked periodically with a cron job or so. You can examine your own mongo by using approach described here http://joey.aghion.com/listing-mongodb-collections-by-size/ to figure out which collection takes what space.

Details

Here are the details I have found while running my own parse servers.

_PushStatus

_PushStatus collection is growing indefinitely adding a record for every push notification sent. It seems safe to prune the records periodically as their only use is for parse dashboard, but it's currently undocumented when is it safe to remove the records to not break anything, especially the push adapter. I'm currently pruning the status via afterSafe hook once the push is sent (or fails to get sent) and it seems to work nicely:

Parse.Cloud.afterSave('_PushStatus', (req, res) => {
    const obj = req.object
    const x = obj.get("status")
    if (x === 'succeeded' || x == 'failed') {
        Parse.Object.destroyAll([obj], {
          success: function() {
          },
          error: function(error) {
            console.log("[afterSave] _PushStatus status '" + x + "' Parse.Object.destroyAll failed with error code: " + error.code)
          },
          useMasterKey: true
       })
    }
});

_User

__User collection is growing with every new user being added. Every record contains when the user was created and last modified but it does not seem to indicate when the given user was actually last used to perform any API call thus making it impossible to remove inactive users after say 3 years of inactivity. It would be great if there is any easy way to find out inactive users and clean them up.

_Installation

_Installation collection is growing with every new app instance started or reinstalled on a new device. Again we know when the installation was created and updated but I fail to see a way to clean up installations that were not used say for 1 year. More over installations are by default not bound to a user so it might get problematic to associate the installation rows with users.

_Session

_Session collection is growing with every new device successfully authenticating against user name and password. Sessions are associated with users so we can see how many sessions each user has and when they were created but we can not clean up sessions that are not being used. I know there is some cleanup mechanism in place for expired sessions (and old sessions after password gets changed). But since sessions do not extend their expiration automatically every time when being used, I'm currently using sessions with expiration time of 5 years to avoid unexpected sign out of my users and thus my _Session collection grows and grows and probably contains lot of unused objects.

Suggested Fix

I'd love to see those turned into actual bug reports and pull requests to make parse server self cleaning, but as a first step it might be good to come up with a set of best practices and fragments that people can use and evolve these over time.

[woodardj]

We use a mongo TTL index on _created_at to keep the _PushStatus collection in check.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[mtrezza]

Interestingly, I see a TTL index for the _PushStatus collection with expiration 30 days after _created_at in all our Parse DBs. So _PushStatus seems to get cleaned up by default.