STRATEGIC PLANNING: "manage" tools and component loaders

[dustymc]

The Issue

Arctos offers two "update results set" mechanisms, component loaders and the real-time updaters. Component loaders are scalable and sustainable, and the real-time loaders are less so. Component loaders can do everything real-time loaders can do, but with a time delay. Any changes would not automatically imply anything at all in the UI.

Component Loaders

"Component loaders" are an ecosystem of bulk tools, managed under a central dashboard (including some user-accessible settings) designed to basically do anything, no matter how long it takes. These are designed with postgres locking mechanism in mind, and generally run quietly behind the scenes without causing disruption. They all run on one row of data in a table at a time, and nearly any mechanism could be used to populate the underlying tables.

Real-Time Loaders

These were all developed in Oracle, and when Arctos was much smaller. Oracle is a 379 billion dollar company largely because the database is very good at negotiating locks; many users trying to do the same things at the same time is generally handled elegantly. Postgres is in comparison primitive; it is easy to lock things, and un-locking often cascades out in strange directions.

Code and Maintenance

There is some (relatively minor) cost to maintaining two sets of tools which do about the same things. The UI is also more complex than it needs to be, likely leading to users not making the best choices, and it is difficult to focus development across mostly-redundant forms.

The Vague Proposal

Unless significant resources (eg Oracle and an Oracle DBA) can be made available, I think it makes sense to deprecate the real-time tools. I'm not sure all of them map exactly to current needs or component loaders (see above, they are all fairly old), but it should be straightforward to convert them to tools which simply write to component loaders. This should have minimal impact on any workflow - things will take some time (very occasionally days, almost always minutes for anything that the real-time loaders will handle) but work much more reliable.

Urgency

I don't think this is too critical from a "keep Arctos alive" standpoint, the database does seem to recover from locks by itself most of the time. It may be much more critical from a usability and user satisfaction standpoint, but those are difficult to judge. It would be useful to at least establish a direction, decide how we're going to focus resources, and begin planning for any changes.

Example

This happened yesterday; one user was using the multiidentification tool, then another use tried to update a single ID, got an error (or didn't, sometimes these kinds of errors aren't reliably passed on to the browser), then everything got all crossed up and there was ~an hour of instability, and I'm sure frustration for the involved users. (Users were sschell and lexibaca, and @ccicero noticed the instability.)

Issues

• Feature Request - Bulk ID Update tool option - update all existing identifications by +1 #7983 (comment)
• Bulk add part attribute fails #7924

[genevieve-anderegg]

So from the user perspective, we would still have functionality with the tools, it would just take a bit longer?
(I know nothing about any of this) Thanks for explaining, I was worried I was going to lose the very helpful tools. If the UI and functionality is the same but the guts of the system are different and it just takes a bit longer, then I can't object if it's better for Arctos overall