Use "extra: EdTechHub.ItemAlsoKnownAs" instead of "archiveLocation" field.

[bjohas]

archiveLocation causes issues with APA and reports - the archiveLocation is printed.

While it feels more fragile to use the extra field, this is already done for some items that do not have archiveLocation.

The proposal is to use a line in "extra" to store the identifiers. I further suggest to prefix this line with "zoteroIdentifiers" rather than "archiveLocation", because from then on, the new line is unrelated to archiveLocation.

[retorquere]

Not just that: the upcoming zotero will migrate stuff out of the extra field if it thinks it has a corresponding "real" field or if it has the name of an existing CSL variable. You really want a clearly unambiguous field that makes it non-zotero. I'd say "zoteroIdentifiers" is still too generic. For BBT I've started using tex.FieldName.

[bjohas]

OK, great - that's important. How about

AlsoKnownAsId

Is that sufficiently cryptic?
Or

EdTechHub.ItemAlsoKnownAs

(as it's to do with the edtech hub plugin)?

[retorquere]

Would that have a key, a title, something else?

[retorquere]

Should I move what is currently in the archiveLocation to the Extra field?

[bjohas]

Yes - what's currently there should be moved. The condition would be: If there's no extra field "EdTechHub.ItemAlsoKnownAs" then move what's in the archiveLocation to "Extra -> EdTechHub.ItemAlsoKnownAs".

Previously

archiveLocation: 123:ABC;456;DEF

now:

archiveLocation: 
extra: EdTechHub.ItemAlsoKnownAs: 123:ABC;456;DEF

Not sure what you mean by "Would that have a key, a title, something else?" ?

[bjohas]

As I think I have provided the needed information, I'm removing the label. Feel free to re-add!

[retorquere]

WRT the migration: what do you want to be the trigger for this? Scanning all the items on start to determine whether any of them need to be migrated is going to be a PITA for the user when the library grows. There is also the problem that if people start using archiveLocation for its intended purpose (whatever that may be) the migration code I would write would always move it to extra.

[bjohas]

Let's just trigger it manually. I.e., it should run when the user runs the plugin. The plugin has very limited distribution (and even for us, only covers a few libraries). Importantly, I'll get everyone to update the code, so that people don't keep putting ids into archiveLocation. In practice, few people do that.

So I would even say we leave this mirgration in the code temporarily. Let's leave it there for say a month, and then remove it.

I think there should be some light pattern matching on archiveLocation before the content is moved. I would suggest that we look whether there's a least one item of the form

1234567:ABCDEF
\d{7}\:[\w\d]{7}

in it.

[retorquere]

The user doesn't really run the plugin. The plugin is installed by the user, but is then ran every time Zotero is started.

[bjohas]

The "add id to archiveLocation" function only runs manually. Items aren't scanned on Zotero start.

[bjohas]

Sorry for accidentally closing!

[retorquere]

So far we've just stored the DOIs in archiveLocation, right? I could test the Archive Location field and corresponding likes in extra to see if they're DOIs (which follow a detectable format with reasonable accuracy), and if so, migrate that to EdTechHub.ItemAlsoKnownAs.

Scanning for a DOI in extra or archivelocation can be done reasonably fast because I can construct a single SQL query that tests for likely presence (probably just testing for "10.") after which I can do a more thorough test on the found items. I use a similar approach for my own plugin. But that all hangs on there being a simple test for likely presence. If the existing use of archiveLocation has things other than DOIs, that may not be tenable.

[bjohas]

So far we've just stored the DOIs in archiveLocation, right? I could test the Archive Location field and corresponding likes in extra to see if they're DOIs (which follow a detectable format with reasonable accuracy), and if so, migrate that to EdTechHub.ItemAlsoKnownAs.

Some item types do not have archiveLocation, so for those we've already used an extra field. I am hoping that this means that we have code that can write to EdTechHub.ItemAlsoKnownAs already. I.e., rather than making that the exception, we make it the rule.

That leaves migration. As we have identifiers in archiveLocation and EXTRA.archiveLocation , those both need to be scanned. The entries should mainly look like:

(\d+)\:([\d\w]+)

bar DOIs, and they could be chained as

(\d+)\:([\d\w]+)\;(\d+)\:([\d\w]+)\;...(\d+)\:([\d\w]+)\;?

I think you could see whether the archiveLocation or extra.archiveLocation contains one combination of (\d+)\:([\d\w]+) (perhaps of the right length for group id and item id). If yes, then the whole thing are identifiers. These should then be moved from archiveLocation and EXTRA.archiveLocation to EdTechHub.ItemAlsoKnownAs.

(I've not really seen archiveLocation used much, which is why I used it in the first place,.... but clearly that was a mistake!)

Regarding DOIs: The DOI - if available - has also been added to the archiveLocation field. I had anticipated that this could be another identifier and as the DOI field wasn't consistent across types, wanted to copy it to the archiveLocation. Maybe that was also a mistake, esp. as the universal DOI field is coming. However, for now, the DOI is part of the item anyway, we don't need to particularly migrate DOIs here: The DOI should already part of what is picked up and what is written to archiveLocation and thus to EdTechHub.ItemAlsoKnownAs. Hope that makes sense.

[bjohas]

I had wondered whether I shouldn't just do the migration with the zotero-cli script? It's a one-off process, and could be done that way. If the migration is hard, then lets just do that?

[retorquere]

Some item types do not have archiveLocation, so for those we've already used an extra field. I am hoping that this means that we have code that can write to EdTechHub.ItemAlsoKnownAs already. I.e., rather than making that the exception, we make it the rule.

That's the easy part.

That leaves migration. As we have identifiers in archiveLocation and EXTRA.archiveLocation , those both need to be scanned. The entries should mainly look like:

(\d+)\:([\d\w]+)

That is not possible with the 2-stage approach I had in mind. The approach I have was:

1. do an SQL query against the zotero database to find candidate items
2. of the candidates, get their details, and examine which are precisely the ones we want

but I can't use regexen in the sql query. I can of course fetch all items and inspect them, but for very large libraries I don't know what the impact would be.

I had wondered whether I shouldn't just do the migration with the zotero-cli script? It's a one-off process, and could be done that way. If the migration is hard, then lets just do that?

If it's truly a one-off, I'd not bother with that. I'd just make a throwaway script that would run in the Zotero javascript runner.

[bjohas]

Yes, it's truly a one-off. Basically, I'd run it against all my libraries, and that should take care of it!

[retorquere]

Then performance also isn't the most important bit.

[bjohas]

No, not at all.

[edtechbot]

🤖 this is your friendly neighborhood build bot announcing test build 0.0.21.187 ("fixes #27")

Install in Zotero by downloading test build 0.0.21.187, opening the Zotero "Tools" menu, selecting "Add-ons", open the gear menu in the top right, and select "Install Add-on From File...".

[retorquere]

The migration script is in https://github.com/edtechhub/zotero-edtechhub/blob/gh-27/eth-migrate-archiveLocation.js

Note that .187 is not yet merged; only after it is merged will it be released as an upgrade.

[edtechbot]

🤖 this is your friendly neighborhood build bot announcing test build 0.0.21.190 ("Update eth-migrate-archiveLocation.js\n\nAs far as I can see, the script has the problem that all archive locations will be moved to extras. However, very occasionally archive locations are used for other things. This isn't the case in the main libraries we use, however, there are many many libraries I have and some of them may have data in the archive location. I've added some pseudo-code above.\r\n\r\nAlso, it may be the case that the script has to run more than once. I've added some pseudo-code to allow that.")

Install in Zotero by downloading test build 0.0.21.190, opening the Zotero "Tools" menu, selecting "Add-ons", open the gear menu in the top right, and select "Install Add-on From File...".

[edtechbot]

🤖 this is your friendly neighborhood build bot announcing test build 0.0.21.192 ("Merge pull request #47 from edtechhub/master\n\nMerge pull request #43 from edtechhub/gh-27")

Install in Zotero by downloading test build 0.0.21.192, opening the Zotero "Tools" menu, selecting "Add-ons", open the gear menu in the top right, and select "Install Add-on From File...".