STRATEGIC PLANNING: "manage" tools and component loaders

[dustymc]

The Issue

Arctos offers two "update results set" mechanisms, component loaders and the real-time updaters. Component loaders are scalable and sustainable, and the real-time loaders are less so. Component loaders can do everything real-time loaders can do, but with a time delay. Any changes would not automatically imply anything at all in the UI.

Component Loaders

"Component loaders" are an ecosystem of bulk tools, managed under a central dashboard (including some user-accessible settings) designed to basically do anything, no matter how long it takes. These are designed with postgres locking mechanism in mind, and generally run quietly behind the scenes without causing disruption. They all run on one row of data in a table at a time, and nearly any mechanism could be used to populate the underlying tables.

Real-Time Loaders

These were all developed in Oracle, and when Arctos was much smaller. Oracle is a 379 billion dollar company largely because the database is very good at negotiating locks; many users trying to do the same things at the same time is generally handled elegantly. Postgres is in comparison primitive; it is easy to lock things, and un-locking often cascades out in strange directions.

Code and Maintenance

There is some (relatively minor) cost to maintaining two sets of tools which do about the same things. The UI is also more complex than it needs to be, likely leading to users not making the best choices, and it is difficult to focus development across mostly-redundant forms.

The Vague Proposal

Unless significant resources (eg Oracle and an Oracle DBA) can be made available, I think it makes sense to deprecate the real-time tools. I'm not sure all of them map exactly to current needs or component loaders (see above, they are all fairly old), but it should be straightforward to convert them to tools which simply write to component loaders. This should have minimal impact on any workflow - things will take some time (very occasionally days, almost always minutes for anything that the real-time loaders will handle) but work much more reliable.

Urgency

I don't think this is too critical from a "keep Arctos alive" standpoint, the database does seem to recover from locks by itself most of the time. It may be much more critical from a usability and user satisfaction standpoint, but those are difficult to judge. It would be useful to at least establish a direction, decide how we're going to focus resources, and begin planning for any changes.

Example

This happened yesterday; one user was using the multiidentification tool, then another use tried to update a single ID, got an error (or didn't, sometimes these kinds of errors aren't reliably passed on to the browser), then everything got all crossed up and there was ~an hour of instability, and I'm sure frustration for the involved users. (Users were sschell and lexibaca, and @ccicero noticed the instability.)

Issues

• https://github.com/ArctosDB/arctos/issues/7983#issuecomment-2293695435
• Bulk add part attribute fails #7924

[genevieve-anderegg]

So from the user perspective, we would still have functionality with the tools, it would just take a bit longer?
(I know nothing about any of this) Thanks for explaining, I was worried I was going to lose the very helpful tools. If the UI and functionality is the same but the guts of the system are different and it just takes a bit longer, then I can't object if it's better for Arctos overall

[dustymc]

This (for parts) was involved in #8342

[campmlc]

I think it would be helpful and generate more informed user feedback to list the real-time update tools that would be affected. I assume we are talking about Search Results: Tools: Manage:Identifications and other similar? I know that having to wait an indeterminate amount of time for component loaders to process when you are in the middle of a complicated multistep workflow is not ideal, so these are not equivalent from a user experience perspective.

[dustymc]

list the real-time update tools that would be affected

I'm mostly looking for acceptance (or rejection) of the idea at the moment. The only specific/actionable use case I'm aware of is in the issue you followed to get here, https://github.com/ArctosDB/arctos/issues/8484.

having to wait an indeterminate amount of time for component loaders to process when you are in the middle of a complicated multistep workflow is not ideal

I think there are about three (four) ways we could go:

1. (parens for terrible ideas: do nothing and deal with the occasional data-threatening system-killing errors; remove mass-update functionality; fail to find focus and scatter our resources; etc.)
2. Attempt to somehow throttle and isolate the real-time tools, which in reality is probably just (1) with complications (PG just DOES NOT deal with concurrent locks very well, updates necessarily involve locks, and real-time tools inevitably find concurrent - still, most of the problems I've seen have involved a substantial number of records)
3. Get tools that will support large blocks of concurrent real-time changes. I know of one such product, it costs ~$50K/CPU/YR (I think we're running 16 CPUs atm but not completely sure) and would need to be accompanied by a competent DBA (they are in high demand, TACC does not have one, this is no small part of why we transitioned to postgres). I don't think this has even hints of 'realistic' anywhere in it, but I can write the code if it does!
4. Use what we have in ways it's happy to exist in, possibly add more "oomph" if performance isn't adequate (see eg https://github.com/ArctosDB/internal/issues/330 - I've been asking for additional resources kinda forever - and do please note that all I can do from here is ask, actually changing things would involve Arctos Governance working with TACC)

[campmlc]

I wholly support number 4 with the emphasis on acquiring additional resources! It seems that resources are the limiting factor in so many things we ask for to make everyone's user experience and Arctos functionality better. I don't know what "resources" are needed or how we get them, and that may be the problem with making this actionable. What do we need to do?