job usage: archive long term job records in the accounting db

[garlick]

As discussed in flux-framework/flux-core#3136, it may be advantageous for flux accounting to do its own job record archival. A suggested approach would be to

1. define table(s) to be added to the accounting db schema
2. develop a python script that uses the job-list interface to request records that have been added since the date of the newest job in the database.
3. the script could be run periodically by flux cron like other accounting scripts

With archival moved to flux accounting, the job-archive module in flux-core could be retired and the flux accounting project can evolve the schema as needed to fit its requirements.

[cmoussa1]

Some other benefits with having flux-accounting be responsible for its own job-archive:

• flux-accounting no longer needs to depend on flux-core's job-archive DB when running the update-usage command when updating the job usage for all users/banks (i.e specifying a path to the job-archive DB when running the command)
• flux-accounting can tailor its use of flux-core's job-list tool to only grab what it needs in regards to job records and store it in a table in its own DB; one less database to worry about!

[cmoussa1]

Thought I would take some time to provide an update as I've experimented with this this week.

I was able to write a Python script that utilizes Flux's job-list and job-info interfaces to fetch the necessary information needed for flux-accounting's job usage calculation (thanks @chu11 and @grondo for pointing me at this comment!). The Python script essentially issues an RPC to get the required attributes from job-list and the required information from job-info and creates a job_record dictionary that holds all of the attributes. Then, after all jobs are fetched and placed into a list of job_record dictionaries, they're inserted into a jobs table that holds each job record, which can be queried by job-archive-interface.py and update-usage while only needing to make minimal changes to both (essentially just changing them to point to the flux-accounting DB instead of flux-core's job-archive DB).

I'm still pretty early on this but wanted to jot a couple of thoughts down at this point:

• the data fetched by job-archive-interface.py could probably use some cleanup. There are some attributes it's currently fetching from the job-archive DB that it probably doesn't need anymore (e.g job-list returns nnodes now), but I think it would make a PR too heavy if I included both the addition of this Python script to fetch new jobs and changing the data fetched in job-archive-interface.py. So, maybe multiple PR's would be good here to 1) transition flux-accounting to using its own job-archive, and 2) clean up data it might not necessarily need.
• this one might be obvious (it wasn't right away to me) that fetching new jobs periodically and adding it to the flux-accounting DB will grow the DB incredibly large. And since job-usage and fair share (by default) only consider jobs up to a certain point, perhaps a purge_old_jobs () function should be considered to be added (side note: I think flux-core is considering different methods of storing old jobs "forever" in some sort of historical storage, so it wouldn't be flux-accounting's responsibility to keep jobs forever at this moment).

[vsoch]

Shameless plug for my PR here, that now three people have needed (and have needed to dig up old comments or otherwise ask for help) flux-framework/flux-docs#229

[cmoussa1]

This should be fixed by #357, so I'll go ahead and close this. I can open more specific issues if any problems arise.