sharing locality edit access: discuss

[GoogleCodeExporter]

Please review the Arctos locality sharing model. 

Current model: Only users who have access to all collections which use a 
locality may edit the locality. For example, if a locality is used by specimens 
in CollectionX AND CollectionY, only users who have access to BOTH CollectionX 
and CollectionY may edit the locality. This completely prevents "not-us" users 
from editing "our" localities; only users who have explicit access to a 
collection may edit localities used by that collection. When a shared locality 
(specimens may be attached to any locality) is found to be incorrect, it must 
be cloned into a non-shared locality before it may be edited. This increases 
complexity and workload, prevents sharing effort, and denormalizes locality 
data (eg, making future corrections much more difficult), but it is completely 
safe.

Possible alternative: Allow all manage_locality users to edit all localities. 
This is NOT a "safe" solution in that "not-us" users can mess up (or fix) "our" 
localities. I see little indication that localities are regularly mangled by 
edits. The vast majority of locality problems seem to come from data entry (in 
which spatial data are usually treated and viewed as text), not from the Arctos 
editing screens (which have various maps and service checks). I believe a more 
open approach would significantly increase the usability of Arctos and the 
overall quality of data at minimal (but non-zero) risk.

This general issue keeps showing up in my inbox, and I think it's generally a 
good idea at least worth formal consideration. It is very much not my domain; 
there are no technical implications to whatever decision ya'll make. Whatever 
we do (if anything) needs to come from the folks who "own" the specimens.

Changing the locality/VPD model is trivial.

We could possibly add notifications of changed localities, but that would be 
nontrivial and is very likely to be overwhelming.

This should probably be presented to the entire Arctos group before 
implementation, should the AC approve and wish to adopt an alternative model. 

Original issue reported on code.google.com by dust...@gmail.com on 8 Apr 2015 at 7:08

[GoogleCodeExporter]

I vote for the alternative model of allowing all manage locality users to edit 
all localities.  This is particularly necessary in the case of our recent 
ability to link hosts and parasites - they should of course share a locality 
and collecting event, and yet specimens are not managed in the same 
collections. Under the current model, if I use the locality ID of a host in a 
different collection in cataloging a parasite record, then neither I nor the 
manager of the host collection can subsequently edit the locality unless we 
both have reciprocal access to each others' collections.

Perhaps we could add an agent/date field to indicate who was the last person to 
edit a locality, in case another user has questions on what was done and why.

Original comment by campb...@carachupa.org on 16 Jul 2015 at 3:25

[GoogleCodeExporter]

See https://groups.google.com/d/msg/Arctos/8tAGdoG-PME/SyE2n1_09-4J

Original comment by dust...@gmail.com on 20 Jul 2015 at 3:49

[ccicero]

I support the idea of opening up 'manage locality' so that anyone can edit a locality, regardless of which collections they 'own', for the reasons given and especially if we want to encourage/increase cross-collection relationships (especially across institutions), which is one of the great strengths of Arctos. BUT I think it's important if we do that to track the 'last edited by' agent and the date. If we open up permissions for editing localities, can we still restrict permissions for editing collecting events so that we don't have other users changing verbatim localities (and thus we can be sure to keep that record of the original verbatim locality)?

[dustymc]

I think it's important if we do that to track the 'last edited by' agent and the date

From the perspective of localities, that is my intent (and to keep the old values as well). From the perspective of things that use locality, I can't guarantee a clear path - there are any number of ways to (dis)associate localities and the things that use them (eg, edit then add specimens vs. add specimens then edit vs. a little of both vs....).

can we still restrict permissions for editing collecting events

We could, but it would take some (minor, I think, but not positive at this time) development. From the non-technical side of things, that may deserve its own discussion - does sharing localities and not events adequately address the bigger problems? (Seems like a significant improvement to me!)

@DerekSikes

[campmlc]

Not sharing events is problematic from the parasite/ host standpoint. Is
there a way to resolve?
On Jun 7, 2016 1:59 PM, "dustymc" notifications@github.com wrote:

I think it's important if we do that to track the 'last edited by' agent
and the date

From the perspective of localities, that is my intent (and to keep the old
values as well). From the perspective of things that use locality, I can't
guarantee a clear path - there are any number of ways to (dis)associate
localities and the things that use them (eg, edit then add specimens vs.
add specimens then edit vs. a little of both vs....).

can we still restrict permissions for editing collecting events

We could, but it would take some (minor, I think, but not positive at this
time) development. From the non-technical side of things, that may deserve
its own discussion - does sharing localities and not events adequately
address the bigger problems? (Seems like a significant improvement to me!)

@DerekSikes https://github.com/DerekSikes

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#729 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AOH0hD06rzi8wXlAu_Ey0H8eAdk26do_ks5qJc1-gaJpZM4IwSF7
.

[dustymc]

Not sharing events is problematic from the parasite/ host standpoint. Is there a way to resolve?

I can't think of any technical solutions - sounds like a social problem (eg, how do we get people to follow instructions?).

[campmlc]

I agree with Carla's point about not changing verbatim localities. But we
still need to be able to share collecting events between parasites and
hosts. Is there a way to limit changes to the verbatim locality of a
collecting event?

On Tue, Jun 7, 2016 at 3:45 PM, dustymc notifications@github.com wrote:

Not sharing events is problematic from the parasite/ host standpoint. Is
there a way to resolve?

I can't think of any technical solutions - sounds like a social problem
(eg, how do we get people to follow instructions?).

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#729 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AOH0hCwOVORbuyl6AG55eXnGjygohal7ks5qJeaOgaJpZM4IwSF7
.

[dustymc]

Yea, if you can come up with some logic that can be written as triggers (or whatever) it could (probably) be implemented.

If you can share without the capacity to update verbatim locality, can you share under something like the current model (eg you can change stuff if you have access to everything that uses it)?

[campmlc]

That sounds reasonable . . .Carla?
I would think we would at least need a big red warning popup anytime anyone
tries to alter verbatim locality.
Here is a potential (fictional) scenario:

MVZ Bird: host of Rausch cestode cataloged at MSB Para.
MVZ Bird verbatim locality = "Alaska" or "unknown" (based on a tag with
only the RLR number)
MSB Para verbatim locality: Fairbanks (based on the Rausch ledger)
What permissions should parasites have to update the MVZ host record?
{arasites can share the collecting event and edit it entirely? Parasites
can share ce and fix specific locality but not touch verbatim? Other?

Here is an actual example of a parasite/host record (MSB Para, MVZ Birds)
with slight differences in higher geog, verbatim/specific localities
(collecting events not shared). In this case, everything is close enough,
but if the host collecting event is updated the parasite record will not
change. Ideally, both should share collecting events so that an update
applies to both.

http://arctos.database.museum/guid/MSB:Para:2252
http://arctos.database.museum/guid/MVZ:Bird:141334

On Tue, Jun 7, 2016 at 5:11 PM, dustymc notifications@github.com wrote:

Yea, if you can come up with some logic that can be written as triggers
(or whatever) it could (probably) be implemented.

If you can share without the capacity to update verbatim locality, can you
share under something like the current model (eg you can change stuff if
you have access to everything that uses it)?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#729 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AOH0hHardLsnr8D-LuZkjFWVTJEYWkLFks5qJfq9gaJpZM4IwSF7
.

[dustymc]

{arasites can share the collecting event and edit it entirely?

That's certainly my preference, but it puts a lot of faith into other collections NOT letting untrained students do crazy things and ....

Parasites can share ce and fix specific locality but not touch verbatim?

That's where we end up if we fully share localities but keep events the same as they are.

http://arctos.database.museum/guid/MVZ:Bird:141334 has what looks like a very good georeference from a very reliable source, while http://arctos.database.museum/guid/MSB:Para:2252 has quad information but lacks coordinates. I wouldn't describe "Alaska" (~650,000 mi^2 of extreme habitat variations) and "Bethel Quad" (~6,000 mi^2 of swamp) as "slight differences," so it seems very useful to merge those data and keep them from drifting back apart. I think building tools to make that easy in a fully-shared model would be straightforward/worthwhile.

If we're not fully sharing, I don't think such a tool would get much use/be worth building.

There are currently 30891 "parasite of" relationships which refer to a host by an Arctos GUID, and 7885 of them do NOT match - at LEAST a quarter of our verbatim localities are not verbatim (unless somehow we're not sharing field notes or there's some other explanation for that). I'll take that as strong evidence that we really need to be sharing, where at least twice as many collections have an interest in correcting found mistakes.

Data for all guid-based relationships with mismatched verbatim locality are at https://docs.google.com/spreadsheets/d/18puv_PGOzctTOBgiKAlRQELp3uiMYAftP5-1vWv0lTs/edit?usp=sharing. Note that anything with a reciprocal relationship will be duplicated, some of these things (eg, parent/offspring) would not be expected to share verbatim locality, and a bunch of the catalog numbers look old (so pre-Arctos = pre-verbatim-vs.-corrected). A lot of what's left is (probably) "close enough" but there are a fair number of things that are appreciably different for no apparent reason.

@DerekSikes

[dustymc]

AWG consensus: implement history, allow shared access, provide access to history and send notifications, AWG will review and further consider #745

[dustymc]

There is a locality archive table, a trigger to maintain it, and a locality history view app running in test. Edit any locality then click "archive" at the top of the edit form, or http://arctos-test.tacc.utexas.edu/info/localityArchive.cfm?locality_id=1144847,1145777 which contains the history of two localities, one with what I hope is more edits than we'll ever see in the wild. Feedback greatly appreciated.

Next step: email alerts.

[DerekSikes]

Dusty,

I was just browsing this
http://www.gbif.org/newsroom/news/new-darwin-core-spreadsheet-templates

and noticed these DwC terms:

http://rs.tdwg.org/dwc/terms/#georeferencedBy
http://rs.tdwg.org/dwc/terms/#georeferencedDate

which I really wonder, again, why we aren't using in Arctos?

I envision a system that tracks georeferences for a place name the same way
we track sci-name identifications for a specimen. Who, when, how, what, and
a history of those as new ones are added over time to the same record. (who
did the georeference, when did they do it, how did they do it, and what is
the georeference)

We record how the georeference was done (protocol) and what it is
(obviously) but not who did it or when.

I've explored your test archive locality table - nice functionality!!! It
records the userID of who made edits to the record.... but we still seem to
missing a field to record who did the georeference and when.

Anyhow, great addition to a shared locality table...

-Derek

On Thu, Oct 13, 2016 at 2:53 PM, dustymc notifications@github.com wrote:

There is a locality archive table, a trigger to maintain it, and a
locality history view app running in test. Edit any locality then click
"archive" at the top of the edit form, or http://arctos-test.tacc.
utexas.edu/info/localityArchive.cfm?locality_id=1144847,1145777 which
contains the history of two localities, one with what I hope is more edits
than we'll ever see in the wild. Feedback greatly appreciated.

Next step: email alerts.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#729 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AIraMwjlHa2WIZdq6PP9PAf8U8W5hFhtks5qzrZxgaJpZM4IwSF7
.

+++++++++++++++++++++++++++++++++++
Derek S. Sikes, Curator of Insects
Associate Professor of Entomology
University of Alaska Museum
907 Yukon Drive
Fairbanks, AK 99775-6960

dssikes@alaska.edu

phone: 907-474-6278
FAX: 907-474-5469

University of Alaska Museum - search 347,746 digitized arthropod records
http://arctos.database.museum/uam_ento_all
http://www.uaf.edu/museum/collections/ento/
+++++++++++++++++++++++++++++++++++

Interested in Alaskan Entomology? Join the Alaska Entomological
Society and / or sign up for the email listserv "Alaska Entomological
Network" at
http://www.akentsoc.org/contact_us http://www.akentsoc.org/contact.php

[dustymc]

In a word, normalization.

GBIF (DWC) is completely denormalized - "Alaska" is repeated for every Alaskan "record" (=Occurrence, which is again denormalized from an Arctos "record"). If we do that, or something closer to it, it'll mean having more copies of the data - more stuff to update (or fewer specimens benefiting from updates) when you find a mistake, georeference a locality, etc.

It's basically a question of what a locality IS, to which there are two obvious possibilities (and some middle ground):

• If localities are shapes, then there's only one THERE (which may be defined by coordinates, error, depth, elevation, polygons, etc.) and the "whodunit" stuff has nothing to do with the shape itself - it doesn't matter who created the shape or how they did so because it's a real unambiguous place. Our model approximates this, but still contains some denormalizers (locality remarks, various units of measure, specific locality, etc.). The only ambiguity is in whether the shape is appropriate for any given use - was the specimen actually collected from within it? Like all determinations, that does come with who/when/etc. data (which we store in specimen_event, the link between specimens and places).
• If localities are more like metadata of the specimen - if you saying THERE and me saying THERE, or you saying THERE today and tomorrow, or two models of GPS asserting THERE, or .... are different THINGS - then we have more than unambiguous shapes, and those data should be stored much closer to the specimens - there's no reason to tolerate expensive queries and complicated code if the data won't let those things do anything useful for us.

I don't think there's any "correct" approach, and I don't really have any strong preferences one way or the other. I do have a list of things that we can't stop doing, but they're all probably possible in a less-normalized model. (The stickiest bit might be between parasites and hosts - those data really need some sort of strong linkage to prevent divergence.)

We currently map specimen_event.determiner (the person who hooked a specimen to a place) to DWC:georeferencedby and specimen_event.determined_date (the date that linkage was established) to DWC:georeferencedate. (And I dislike the term "georeferenced" - it implies that we started with descriptive data, which isn't always true.)

[dustymc]

I need input on email alerts of changed Localities.

We have Localities which are linked to Media, or linked to Media through Events. There are not necessarily any other relationships involved. I need a collection_contact agent if I'm to send email. How to proceed?

@ccicero

[dustymc]

This is now running in prod as https://github.com/ArctosDB/arctos/tree/v7.4. I'm leaving the VPD as-is and this issue open for some real-world testing; please comment here if there are any performance issues editing localities, if you are not getting the proper alerts, or if anything else doesn't seem to be working as it should.

Click 'view edit archive' at the top of edit locality to see archives (or /info/localityArchive.cfm to browse).

If there are no issues, the locality VPD can be disabled and this issue closed after ~2016-11-07

[dustymc]

Is everyone getting useful locality change notifications? Can we drop the VPD restrictions and allow global locality edits now?

[mkoo]

which tables? just locality, right? I'm up for it especially with
georeferencing history, so we need to reinstate GeoreferencedBy and
georeferencedDate

I've just been cloning like a mad sheep...

On Mon, Nov 14, 2016 at 1:21 PM, dustymc notifications@github.com wrote:

Is everyone getting useful locality change notifications? Can we drop the
VPD restrictions and allow global locality edits now?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#729 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACZ_0bwvQ0puqtPGbocACOTUkYO0DKEqks5q-NDdgaJpZM4IwSF7
.

[campmlc]

yes, let's do it.

On Mon, Nov 14, 2016 at 4:04 PM, Michelle Koo notifications@github.com
wrote:

which tables? just locality, right? I'm up for it especially with
georeferencing history, so we need to reinstate GeoreferencedBy and
georeferencedDate

I've just been cloning like a mad sheep...

On Mon, Nov 14, 2016 at 1:21 PM, dustymc notifications@github.com wrote:

Is everyone getting useful locality change notifications? Can we drop the
VPD restrictions and allow global locality edits now?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#729 (comment),
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ACZ_
0bwvQ0puqtPGbocACOTUkYO0DKEqks5q-NDdgaJpZM4IwSF7>
.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#729 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AOH0hLv8fAtG7Kq3TaDXjzvJR9P_r123ks5q-OjqgaJpZM4IwSF7
.

[dustymc]

Yes, just table locality. If you've changed anything in the last ~2 weeks, or are a 'data quality' contact for a collection that's change anything in the last ~2 weeks, you should have email from locality_change@arctos.database.museum.

so we need to reinstate GeoreferencedBy and georeferencedDate

If we do that I'm not sure there will be much left to share. Those data are certainly not in anything I've written.

??

[mkoo]

OK, maybe AWG needs to discuss this.
Someone (Derek I think) remarked about the DwC fields for georeferencedBy
and georeferencedDate are missing, and it's true they are not part of the
locality table where the XYCoordinates are assigned.

Instead the Arctos model has Specimen Event table and an Event Determiner
(curatorial assistant or other )

Do we need to track determiners in both places? The Specimen Event model is
nuanced differently than the DwC model, being much more specific to the
specimen assignment.

Do we want to simply make locality table a wide open geography lookup or
assign the specimen-nuanced localities only in specimen events? This really
requires a bit of a mind-shift I think since it goes against the early
concept of locality/georeferencing interpretation etc.

Is there value for collections to track how XY are determined? or only how
XY are used?

On Mon, Nov 14, 2016 at 3:47 PM, dustymc notifications@github.com wrote:

Yes, just table locality. If you've changed anything in the last ~2 weeks,
or are a 'data quality' contact for a collection that's change anything in
the last ~2 weeks, you should have email from locality_change@arctos.
database.museum.

so we need to reinstate GeoreferencedBy and georeferencedDate

If we do that I'm not sure there will be much left to share. Those data
are certainly not in anything I've written.

??

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#729 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACZ_0Za2TcjAWCOwnqq6aTY94EYuz6oxks5q-PMigaJpZM4IwSF7
.

[dustymc]

I think the fundamental question is whether locality data are their own data objects or metadata of something else (like specimens, but that leaves no media-->locality pathway).

#729 (comment)

[dustymc]

All locality data are now fully shared and may now be edited by anyone with the proper permissions; closing this Issue.