LTA Command & Control Components

[blinkdog]

Jim has some Command and Control components that manage the work fed into the LTA pipeline and the cleanup after archival is complete.

This issue is about deciding and planning to integrate these components into LTA proper, spin them off into an LTA metaproject, etc.

jbellinger 1:16 PM
There are some other components which aren't (yet?) part of the LTA pipeline--the dumped-file-deleter, and the get-a-new-directory-to-request bit

pmeade:scouting: 1:17 PM
What do each of those do?

jbellinger 1:25 PM
My dumper system dumps, checks for completeness, and feeds directories into the hopper. At the end of the LTA, if the dumped files aren't wanted locally, my "dumped-file-deleter" chucks them. The "get-a-new-directory-to-request" is my interface from the hopper to the LTA system; it's fed from a DB I maintain on archivecontrol. That gets directories considered good from the dumper system, and also from anything I manually tell it is good (e.g. /data/exp/IceCube/2017/filtered/PFFilt/1231, which I told it about a few minutes ago).

pmeade:scouting: 1:28 PM
Ahhh okay. For now I'll add a card to the project board about them so we can discuss with more brains.

[jnbellinger]

https://docs.google.com/spreadsheets/d/1bSXpeqq5DC5hTDUyums6jmU_LAtW71F6msT3iDB3eHM/edit#gid=0 LTA_Component_Control
I need tools to tell

1. nersc_supervisor
2. Cron on lta-vm-2
3. Kubernetes
   I need a non-manual starter for several modules.
   Advantages for regular (run once and die: Shutdown is easy; just don’t run any more. You don't interrupt activity and drop bundles on the floor.
   Disadvantage: have to keep track of how many modules are running and where. Maybe you really want to shut things down (power outage due).