Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add cron job for lobid-organisation labels map #1903

Closed
TobiasNx opened this issue Sep 25, 2023 · 8 comments
Closed

Add cron job for lobid-organisation labels map #1903

TobiasNx opened this issue Sep 25, 2023 · 8 comments
Assignees
Labels

Comments

@TobiasNx
Copy link
Contributor

TobiasNx commented Sep 25, 2023

src/main/resources/alma/maps/lobidOrganisationsMapping.tsv

Should be fetched via cron job.

https://lobid.org/organisations/search?q=_exists_%3Aisil+OR+type%3ACollection&format=csv:id,isil,sigel,name&size=25000

The process should fetch this and replace:
^" => empty
"$ => empty
"," => Tab
",," => 2x Tab

It seems that the request is forbidden in prod, but it works on test.

The columns are id, isil, sigel, name

This was asked by Digibib, since the labels shoud be updated regularly.

@dr0i
Copy link
Member

dr0i commented Sep 26, 2023

It seems that the request is forbidden in prod, but it works on test.

I've now increased in the webserver config to timeout=60.

@TobiasNx
Copy link
Contributor Author

thanks this seems to work now.

@TobiasNx
Copy link
Contributor Author

Can you help with the seed script? For the cron job?

@dr0i
Copy link
Member

dr0i commented Sep 28, 2023

Instead of regex the output it would be comfortable to have a tabulator-separated export in lobid-organisations => depends on hbz/lobid-organisations#500.

@TobiasNx
Copy link
Contributor Author

TobiasNx commented Nov 23, 2023

The new mechanism does not need any post processing. Only the following must be done:

  1. Move the local file from src/main/resources/alma/maps/lobidOrganisationsMapping.tsv to src/test/resources/alma/maps
  2. Cron job to create new src/main/resources/alma/maps/lobidOrganisationsMapping.tsv file with https://lobid.org/organisations/search?q=_exists_%3Aisil+OR+type%3ACollection&format=tsv:id,isil,sigel,name&size=25000

@TobiasNx
Copy link
Contributor Author

Also adjust the links for the test here:

fixVariables.put("lobidOrganisationsMapping.tsv", "./maps/lobidOrganisationsMapping.tsv");
fixVariables.put("hbzowner2sigel.tsv", "./maps/hbzowner2sigel.tsv");

fixVariables.put("lobidOrganisationsMapping.tsv", "src/main/resources/alma/maps/lobidOrganisationsMapping.tsv");
fixVariables.put("hbzowner2sigel.tsv", "src/main/resources/alma/maps/hbzowner2sigel.tsv");

@dr0i
Copy link
Member

dr0i commented Nov 27, 2023

lobidOrganisationsMapping.tsv will be downloaded just before the automatical weekly deployment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: Done
Development

No branches or pull requests

2 participants