automated update of resources and index

[SchSascha]

No description provided.

[SchSascha]

• aim: daily medline update
• pmc: look for best solution (really full download each time?)

[SchSascha]

My current request from partners includes to be able to find most current hits (e.g. Sep 2017), which likely includes a recognisable event.
Not only shows this the urgency of an updated data basis but also highlights our need for this issue to be resolved. What's the state of things here? What is the estimated workload? Need to discuss!

[khituras]

Requirements: 1. Regularly download Medline Updates 2. Regularly download PMC completely, because there are no separate updates 3. Process new documents 4. Index new documents 5. Create automatic notification in case of errors All these points require a CRON job, which is not an issue. 1. is already active. 2. is an issue because of the fact that we always need to download PMC completely. Some thought should be done into possibilities to keep the workload as little as possible. We cannot process all of PMC every week or so. 3. Requires shell scripts for a CRON job. I will write these. 4. Like 3. 5. Won’t currently work because of a missing Mail Gateway. Benjamin wanted to set one up for a long time as views our requirements as a good reason to finally do it.

…

On 1. Dec 2017, at 23:45, SchSascha ***@***.***> wrote: My current request from partners includes to be able to find most current hits (e.g. Sep 2017), which likely includes a recognisable event. Not only shows this the urgency of an updated data basis but also highlights our need for this issue to be resolved. What's the state of things here? What is the estimated workload? Need to discuss! — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#40 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEbucLsX-x3RwQ-maW7Ngt5dnNKVy-ALks5s8IGegaJpZM4NTyaj>.

[khituras]

Status PMC: Benjamin needs to give me an update mechanism for PMC because we need to download it completely every time but not every document is new. He knows what to do and should be reminded at a regular basis.

[khituras]

Status PMC: Benjamin gives an "unknown" deadline due to other work. Thus, for the time being we will just have to reprocess completely for updates.
Or, as a dirty workaround, we only add new documents and ignore the possibility that old documents have undergone a change.

[SchSascha]

Unknown sounds alarming.
Adding only new documents is fine for the moment.

[khituras]

Benjamin: Thursday January 23 2018 afternoon he can say whether he can do it or not.

[SchSascha]

He can't can he?
So, what is the latest time line, here, what do you think?

[khituras]

No, he can't. I will do it next. The facts:

• PMC does not offer update packages but is always updated as a whole.
• PMC documents can be downloaded individually but this takes ages.

The plan:

Init:

1. Create a directory A from where PMC documents will be read and processed in the future.
2. Download the latest PMC distribution to A.

Update:

1. Download PMC into directory B.
2. Create a files HA and HB that contains the filename of each document in A and B, respectively, and their MD5 hashes.
3. Compute the differences of HA and HB. Perform appropriate insertions, replacements or deletions from B to A. Create a new file PA with newly inserted or replaced documents, create another file DA for documents that have been deleted.
4. Apply the changes to the PMC archives in A.
5. Delete B if desired.
6. Apply the changes to the database (Postgres).

I will give a report on Monday the 11th on how it went, if there were any issues and how long it will still take. I will be able to judge better then than now.

[khituras]

Update: I began writing the script. I think I will be done by Wednesday with finishing it.

[khituras]

Update: Well, was not quite finished ;-) I fixed some bugs and now all steps until 5. are working. Can't say when step 6 will also be working as the TREC PM 2018 deadline approaches (August 5).

[khituras]

See https://github.com/JULIELab/jedis-pmc-updater for a new repository where I store the respective files and describe them.

[SchSascha]

Is this issue still open?

[khituras]

It is. I have made progression here, but the last step is still missing (having the PMC UIMA reader understand what to (re)process and what not).

[SchSascha]

Appr. time for resolving?

[khituras]

2 days (conservatively).

…

On 18. Oct 2018, at 13:38, SchSascha ***@***.***> wrote: Appr. time for resolving? — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#40 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEbucKbIG148MSxvZzUY2Rz6tsF3qIpEks5umGhDgaJpZM4NTyaj>.

[SchSascha]

OK.
I believe, next to recovering output for another round as input, having full texts results is essential as well.
Hence, I consider the automatic update also as critical, especially with respect to NAR.