Job trimming for monitored tags

[antons-]

From what I can tell, when you monitor a tag in Horizon it does so indefinitely until you stop monitoring that tag or your Redis instance runs out of ram.

We would like to keep monitoring tags over an extended period of time, but have a way of configuring job trimming for monitored tags, just like with recent and failed jobs.

Ideally I would like to trim jobs older than a week automatically with monitored tags.

Is this already possible or is there a different approach here I can take?

[driesvints]

Could be a nice feature to add an input box with the desired minutes on how long to monitor a tag. I'll mark this as a feature request.

[fgilio]

This bite me one time and the whole thing crashed cause Redis consumed all available memory, which was a tone. It'd be awesome to trim jobs older than x amount of time or just keep up to n amount of jobs.

[mfn]

I believe this is (or: related to) #271

[driesvints]

@mfn @fgilio do you believe this can solve #271?

[fgilio]

Yes, I think it can partially solve that issue.
What I mean with partially is that it will depend on the amount of jobs. If a system has 1000 jobs per minute then the Redis db could be filled in just an hour. But this may be solved by just making the value configurable and stating this fact clearly in the docs, so every dev chooses an appropriate value.

EDIT: I think it'd also be useful to warn users that job monitoring makes Horizon keep a record of the job in Redis, and that a lot of jobs might fill up the entire Redis db. So things like #271 doesn't come like a complete surprise.

EDIT 2: Thanks @driesvints for being on top of all this :)

[mfn]

@driesvints I second what @fgilio said above: depending on the amount of activity there seems to be always a possibility to "eat too much ram".

I guess the only real solution would be an implementation of an RRD which usually have fixed sizes, but I digress.

[driesvints]

I've sent in a PR for this: #484

About the problem of speedy fill ups: I think you'll probably have bigger problems in that case and it doesn't really makes sense to monitor anyway if your DB can't keep up.

[driesvints]

@mfn @fgilio can you both have a look at #484 and let me know on the PR itself if it looks ok to you?