Feature Request - agent attributes - phase two

[dustymc]

Is your feature request related to a problem? Please describe.

Agent attributes exist and are running in production. Next step is to move everything else into the unified structure.

Describe what you're trying to accomplish

Unity and happiness.

Describe the solution you'd like

Create attributes corresponding to each item in:

• https://arctos.database.museum/info/ctDocumentation.cfm?table=ctaddress_type
• https://arctos.database.museum/info/ctDocumentation.cfm?table=ctagent_name_type
• https://arctos.database.museum/info/ctDocumentation.cfm?table=ctagent_relationship
• https://arctos.database.museum/info/ctDocumentation.cfm?table=ctagent_status

plus

• agent.agent_remarks
• agent.curatorial_remarks
• agent.preferred_agent_name

Include appropriate controls and rules as necessary.

• make sure shipping addresses remain protected, whatever that looks like in this model
• filter ORCID, wikidata, etc.
• limit (or document?) the attribute correspondening to agent.preferred_agent_name to preferred name change history

Remove:

• agent.last_edit_by
• agent.last_edit_date

The new model will provide a more robust way to track these (along with WHAT was edited).

Drop:

• unique constraint on agent.preferred_agent_name, Use of preferred name as unique key for agent table #4534

I will adjust the UI to follow #6813 and friends while I'm changing things. Primarily, that's no secondary data necessary for agent creation, and no requirements or presumptions of uniqueness on preferred name.

Describe alternatives you've considered

I could open issues for each individual term/action, but I think one issue for all is probably easier to follow.

Additional context

Specific concerns (eg stuff mentioned in #7318) can be addressed as necessary during or after this migration.

Related/Included

• Feature Request - search agent curatorial remark #6699 - will just work in new model
• select agents by more things #7322 - will be released with new model, necessary in an environment with nonunique agents.

Priority

Critical - we are now in an intermediate model with denormalized data and an overly-complex UI, and important data are almost certainly still being stuffed into remarks.

@mkoo @Jegelewicz someone anyone please say GO!

[mkoo]

This is a great consolidation of all the various issues and a nice roadmap to get to our new goal of Agents.
Please start on the tasks in its logical order as it has been approved in AWG and Agent Committee meetings (among other committee meetings)

The only one I dont see in the chain of requirements (create list) is this one (unless it is implicit):
#6813 (comment)
which allows the check on verification/ minimum requirements for agents. This needs to be included

[dustymc]

verification

Sorry, things got a little convoluted. #6813 (comment) became #7318 which is in production now (but not DOING much as it's behind this issue). If there's something about the UI that could be better (other than all the legacy stuff piled on top, of course), please let me know NOW - it'll be the only UI once this is done.

I of course gave myself Attribute One and a gold star.

I'll run this through the dev channel for more exposure and get started.

[mkoo]

Ah cool! Testable in test then??? Can we have a demo of the progress of these tasks at the Issues meeting on Thursday?

As for #6813/ #7318 ideally this is a largely automated process based on specific criteria which I dont know if it has been on the agenda of the agents or AWG committee, but probably should be.

[Jegelewicz]

Testable in test then???

It's in production

[dustymc]

Testable

Yep, in prod.

demo

Doesn't do much yet, just the model and UI made it through the Phase One process. But yea, if someone wants to complain about the UI (or start planning for a replacement - I think @Jegelewicz has ideas) or point out something that the model could but does not do or etc now's the time.

MAYBE I'll have something more in test by then, but probably going to be all behind-the-scened for a few days. I'm working on https://arctos-test.tacc.utexas.edu/info/ctDocumentation.cfm?table=ctagent_attribute_type now.

largely automated process

No problem, just file an Issue with the criteria (after this) and I'll set up a bot, or periodically script things in, or whatever turns out to be best.

[Jegelewicz]

if someone wants to complain about the UI (or start planning for a replacement - I think @Jegelewicz has ideas)

I hadn't thought about it, but since I can see what you have. Let me propose something.

Can we make the agent search have the same look and feel as the catalog record search? Search form at the top, results at the bottom - this could let you see and possibly compare results to make better choices. So each row of the results would have something like this:

preferred name	given names	family names	born	died	ORCiD	Wikidata	Library of Congress	employment
Teresa J Mayfield-Meyer	Teresa	Jegelewicz; Mayfield-Meyer	1800-01-01		https://orcid.org/0000-0002-1970-7044			Collection Manager at University of Texas - El Paso; Community Coordinator at Arctos Consortium

Sorting by column headers would allow for easier comparisons as would the ability to check one or more and only see those (remove some rows like you can in catalog record search).

Editing would be a separate page, just like a catalog record. So from the results, if I wanted to edit my Agent record, I would click my name and end up at the edit page (or the public profile if I am a public user).

This means we should also think about how or if the search should be different for public users and editors. I don't think it needs to be? Just hide the appropriate attributes from anyone without manage_agents?

Next up - organizing the edit page.

[dustymc]

A full-width page would certainly work better for editing.

They data aren't very tabular, I don't think your example is realistic, but I can figure some sort of less-bare results out (and it'll be easy to modify as we find things that could be better).

remove some rows

Seems like wild overkill (that sort of thing is hugely complex), but see above - if we need to we can.

organizing the edit page

[Jegelewicz]

This is the current demo

Once all of my information is in that format, it will just be a long list of hard to process double rows - however, if we weren't taking up half of the edit page with a blank white space because the search form is over there somewhere at the top, we might be able to fit each attribute on a single row - I think that would be a good start to making this less hard to deal with.

Still a long list of stuff without any breaks is hard on the eyes and the mind. If possible, it would be nice to organize the entered attributes by the broad categories - this would be similar to the current edit page and one familiar thing would be good.

I also think that the section for adding new information should be at the top of the edit form. If it starts with three rows (and has an add more) it won't be too intrusive and if I am coming to add a new attribute, I don't want to scroll to the bottom every time.

Manage Agent

Add Attributes

attribute define	attribute value	begin date	end date	related_agent	determined date	determiner	method	remark
add row

save

Edit Attributes

save

Names

attribute define	attribute value	begin date	end date	related_agent	determined date	determiner	method	remark
given name	Teresa				2024-02-02	jegelewicz	personal knowledge
family name	Jegelewicz				2024-02-02	jegelewicz	personal knowledge
family name	Mayfield-Meyer				2024-02-02	jegelewicz	personal knowledge

Addresses

attribute define	attribute value	begin date	end date	related_agent	determined date	determiner	method	remark
ORCiD	https://orcid.org/0000-0002-1970-7044				2024-02-02	jegelewicz	personal knowledge

Milestones

attribute define	attribute value	begin date	end date	related_agent	determined date	determiner	method	remark
employment	Collection Manager			University of Texas - El Paso	2024-02-02	jegelewicz	personal knowledge
employment	Community Coordinator			Arctos Consortium	2024-02-02	jegelewicz	personal knowledge

Statuses

attribute define	attribute value	begin date	end date	related_agent	determined date	determiner	method	remark
status	verified	2024-02-01			2024-02-02	jegelewicz	personal knowledge

save edits

[dustymc]

double rows

That's up to your browser.

broad categories

That's why I just added that to the code table....

adding new information should be at the top

Always has been??

[mkoo]

Not sure if you have another issue open for the test version of the agents manage/ edit/ public interface but some general comments. Overall I think it works but test was testy so havent really been able to try out everything I wanted to.

One thing I think needs to be changed on the public page is to have the remarks right under the top line of the name. That is where we have been adding all our biographical info (and its been a lot since we are importing our finding aid bios) so should be up top with the name and not midway or bottom half of page (save that area for the nitty gritty)

I'll return to this when test is more cooperative!

[dustymc]

another issue

Nope, here's good, and thanks!

the remarks

Details, probably, but: There is no THE, there can be 987 "remarks fields" (or anything else) in this model, and they can all carry (maybe critical) metadata - nothing really survives this in 'just a string' form. Also....

adding all our biographical info

... a place dedicated to that is super-trivial (just a code table request), maybe we should do that, move some stuff, and make that (rather than the 'junkdrawer data' remarks is meant to hold) the prominent thing? That'd have to be post-release, happy to help shuffle, @AJLinn might want in on that conversation.

test is more cooperative!

It should be happy now. It is slow, but I think that's just test - I tried bits individually at prod and they're fast there. It'll also be different based on who you are - eg https://arctos-test.tacc.utexas.edu/agent/280 will be the agent detail page (possible because of how I structured https://arctos-test.tacc.utexas.edu/info/ctDocumentation.cfm?table=ctagent_attribute_type). Similarly https://arctos-test.tacc.utexas.edu/agent.cfm will be the search page, users with different rights will just be able to access different things.

[Jegelewicz]

So search will now be unified? This seems odd

I would expect my agent to show up in results

[Jegelewicz]

But the "new" isn't using old data yet?

This looks good

[Jegelewicz]

OK so the search name thing is using "starts with"

I think the old search was "any part of any name". I guess everyone would get used to it, but that single change might make a lot of people upset?

[Jegelewicz]

I would remove the admonishing frowny face here. I don't want people knowing my birth date - after I'm dead you guys can add it, but for now I'd like to keep that to myself if possible....don't make me feel bad about that.

[Jegelewicz]

Buttons at the top that let me scroll to one of the sections? Like at the top of the catalog record page.

also here

[dustymc]

fix

It's not broken, it's #5172 - this addresses a stated need for more functionality, it's not just different UI, and it's not the same capabilities as the old UI. "There, doing that, with them, then" was not possible before and is now. That requires a different UI.

relationship

UI and triggers are different; I can't envision a situation in which value and agent should differ, and hopefully that's not just a lack of imagination regarding the above.

[ebraker]

It's not broken, it's #5172 - this addresses a stated need for more functionality, it's not just different UI, and it's not the same capabilities as the old UI. "There, doing that, with them, then" was not possible before and is now. That requires a different UI.

I'm not saying it's broken. Simply that big changes can frustrate people so we need to have some sort of basic visual decluttering of this page. Yes we are gaining functionality, but if no one uses it because it is too intimidating/overwhelming, then we have lost something. I don't think it would take that much to make the form a little less unwieldy.

[DellaCHall]

Overall, I am a fan of this change - thank you! I have a few questions/suggestions for improvements:

1) Agent auto-populate issue: I used to be able to type a name exactly and it would just populate into any agent/determiner field, if there were no duplicates. Now it seems to search every field, including relationships, so all people/organizations associated with an agent also show up, then I have to click the correct one. This added extra step seems silly when typing a full name exactly, and might actually deter people from adding relationships, due to the annoyance (for example, I'll probably remove my relationship with my husband, so I can enter data faster). Is there a way to have it auto-populate the agent if there are no duplicates in the name fields, like it used to? I'm not sure what is gained by this particular change, so maybe I just need a different perspective to understand it.

2) Search Results Table: Could the search result table headers be customizable like the record search page? I'm not sure why those particular fields are the default headers, but they're not what I would choose. Also, could this table of data be exportable? Could it be sortable? Especially by name types? I have always disliked that the data sorts automatically by first name. As others have mentioned, right now it's just a really long list to scroll through...not very usable.

3) Remarks field: I see there was already some discussion with @mkoo about biographies that have been previously stored in the remarks field. Hoping there will be an easy way for me to migrate that data to the new biographies field (which I like!). Remarks currently show up in the search results table under "Other", but not on an Agent's individual summary page. Can Remarks display on the agent summary page, too? Previously it displayed.

4) Verbatim agent: I think the search question of "Include verbatim agent?" should default to yes instead of no.

5) Hopeful question: Is this a first step in reintegrating verbatim agents at some point down the line?

Editing to add one more thing:

6) Used by Collection: This field was previously an option on Manage Agents as a search field, and it appears to be missing from this new iteration. I liked that field, can it be added to this?

[ewommack]

Agent auto-populate issue: I used to be able to type a name exactly and it would just populate into any agent/determiner field, if there were no duplicates. Now it seems to search every field, including relationships, so all people/organizations associated with an agent also show up, then I have to click the correct one. This added extra step seems silly when typing a full name exactly, and might actually deter people from adding relationships, due to the annoyance (for example, I'll probably remove my relationship with my husband, so I can enter data faster). Is there a way to have it auto-populate the agent if there are no duplicates in the name fields, like it used to? I'm not sure what is gained by this particular change, so maybe I just need a different perspective to understand it.

Try entering your user name instead of your full name. That works for me to only pull up me. With the relaxation of allowing people to have the same name now we're going to hit into having two or even more of the same person coming up in an exact name search as this as we get more and more people with the same or very similar names brought into the system. I think it is probably safer to have us search for people unless you know exactly who you want.

[Jegelewicz]

Is this a first step in reintegrating verbatim agents at some point down the line?

@DellaCHall yes it is.

[Jegelewicz]

I'll probably remove my relationship with my husband, so I can enter data faster

NOOOOOOOO! Do what Beth says and use your Arctos username.

[Jegelewicz]

why those particular fields are the default headers

They are the groups of attributes as they currently exist in the edit form (mostly)

they're not what I would choose.

What would you choose?

could this table of data be exportable?

Probably - @dustymc could say

Could it be sortable? Especially by name types?

Name types are not one of the columns, so maybe this is something that would come from my question above. But now, people could have more than one "first name" attribute, so this may not be easy to handle.

[Jegelewicz]

I think the search question of "Include verbatim agent?" should default to yes instead of no.

This is probably one of those 50/50 decisions - half will want it default yes and half no and we have no way to please everyone.

[Jegelewicz]

Used by Collection: This field was previously an option on Manage Agents as a search field, and it appears to be missing from this new iteration. I liked that field, can it be added to this?

@dustymc possible?

[DellaCHall]

Try entering your user name

I appreciate the suggestion of using my Arctos username! I can certainly do that for myself. And I won't remove my relationship to my husband, @Jegelewicz :)

I think it is probably safer to have us search for people unless you know exactly who you want.

Sure, but when I do know exactly who I want, I think I should be able to type their name and get just them (so long as there are no duplicates of the name fields). This is a super basic function many databases have (and, Arctos currently has!).

having two or even more of the same person coming up in an exact name search as this as we get more and more people with the same or very similar names brought into the system

I understand. But these aren't in the name field of the agent - they're not even close to the same names. As an example, the agent Alaska Department of Fish and Game has a lot of "associate of" relationships. One of those is someone named "Doreen I. Parker McNeill". If I type in "Doreen I. Parker McNeill" as a determiner on an agent record attribute, why should it pull up a big list of ADF&G associates I have to scroll past, just to get to what I typed in to click on it? I stand by my suggestion that agent fields should auto-populate (as they do now) if there are no duplicates in the name fields, specifically.

[Jegelewicz]

these aren't in the name field of the agent - they're not even close to the same names. As an example, the agent Alaska Department of Fish and Game has a lot of "associate of" relationships. One of those is someone named "Doreen I. Parker McNeill". If I type in "Doreen I. Parker McNeill" as a determiner on an agent record attribute, why should it pull up a big list of ADF&G associates I have to scroll past, just to get to what I typed in to click on it? I stand by my suggestion that agent fields should auto-populate (as they do now) if there are no duplicates in the name fields, specifically.

Can we just eliminate related agents from the pick searches? I think that would do it.

[DellaCHall]

They are the groups of attributes as they currently exist in the edit form (mostly)

Okay, I see now. I was confused because I didn't see the new Bio field included...But I'm now seeing Profile, grouped under Other.

What would you choose?

It would depend on the search I was trying to do and the specific data I needed, hence my request to customize/export/sort. That said, something useful for me would be splitting the names out first, middle, last (particularly for sorting, if that could be an option), and all the AKAs could group together. I'm not sure if everyone else would want that, but it would be extremely useful for me.

[dustymc]

I think there are at least two new Issues in here.

1. The pick can do whatever, there are functional implications to those things. (The unified format makes deep digs easier, I think I probably did whatever's there as an effort to reduce duplicate creation. I think that's deserving of dedicated discussion, clearly there's some difference in viewpoints and that's a globally-shared form.)
2. The old search returned a list of names - new does that (and some more), I think the details would benefit from more focused discussion than we're likely to achieve down here.

[dustymc]

Agent auto-populate issue

#7434

Search Results Table

New Issue, I don't understand enough of your comments to kick that off. At this time it's just a simple table without much room to customize (but it's far from a list and never sorted by first name so maybe we're not talking about the same things??). I think @mkoo has maybe-not-immediate API-involved plans, that should be an opportunity to do WHATEVER.

download

#6890

Hoping there will be an easy way for me to migrate that data to the new biographies field

There's CSV in #7418, I'll update at release, PLEASE join in.

Verbatim agent: I think the search question of "Include verbatim agent?" should default to yes instead of no.

#5157

reintegrating verbatim agents

Yes, this will allow low-information agents - but I still hope @ArctosDB/agents-committee will provide some guidelines, and this certainly won't change my recommendations regarding verbatim agents. Lots of discussion in #7135.

Used by Collection

done

[dustymc]

Stuff from https://docs.google.com/document/d/1lXFhBd4LH4802pOEGkj892ZUcJt19kh8Os1ky-qG570 implemented

[Jegelewicz]

I don't think we communicated about the colors well.

The Manage bar is dark blue - but what I expected was that address, event, other and relationship bars would be dark blue so that it is easy to see when you have moved from one "type" of attribute to another.

I do like how the scrolly keeps the top of the form limited!

And the single lines are so much easier on the eyes!

[dustymc]

bars would be dark blue

Crayons please, you've lost me. (Or we've come full circle and you're asking me to put back the thing that I was asked to remove a couple days ago...)

[Jegelewicz]

crayons please

[dustymc]

oh my...

[Jegelewicz]

That's OK - it makes it pretty clear that this agent has very limited data....

[DellaCHall]

it's far from a list and never sorted by first name so maybe we're not talking about the same things??

Maybe I didn't use the right words - it appears to me to be sorted by preferred name? Preferred name automatically sorts by the first letters in that field, which are written with the first name first, hence my saying it sorts by first name.

Search Results Table

New Issue, I don't understand enough of your comments to kick that off.

The gist of it is that it would be extremely useful to me (and maybe others) to be able to customize and/or export and/or sort the data that displays in this results table differently (like sorting by last name, for example). If that's a New Issue (or multiple New Issues) to be tackled in the future, cool! Thank you.

[ewommack]

The DEI Committee met today and reviewed the new Agent and Agent Attribute system. Overall we think that the new systems will be a step forward to opening up Agents to be more culturally inclusive.
We have four suggestions:
#7437
#7439
#7133
and
#7440

[dustymc]

I nuked a bunch of full duplicates and made a sorta-unique index:

create unique index iu_compound_agent_attribute on agent_attribute (
    agent_id,attribute_type,md5(attribute_value),begin_date,end_date,related_agent_id
) nulls not distinct where deprecation_type is null ;