Operationalize MDC: Create Cron Jobs, Acquire, Configure Prod Web Token, Handle Logs

[kcondon]

The MDC feature is well documented but there are a few items that need to be addressed to operate in a production environment:
-Create Cron job(s) to cal the various API endpoints needed to process various files, import to db, including error detection and notification of failure
-Acquire and configure a production web token that allows publishing stats to DataCite
-Consider/Plan/Monitor growth of log files
-Consider how to troubleshoot or rerun failed jobs.

[djbrooke]

I'll add one more...

• we'll need to figure out how to handle pre-MDC download counts. I'd like to reflect them so that researchers don't need to start at zero. :)

@dlowenberg @mfenner it would be good to get some thoughts from you and the team on how other groups have handled this. Thanks in advance for any guidance or for pointing us to any docs!

[dlowenberg]

Hi there, if you would like to look at or copy the code that we wrote for Dryad in processing the last ten years of downloads, here is some info that may be useful:

The main reporting code is here: https://github.com/datadryad/dryad-repo/blob/dryad-master/dspace/modules/api/src/main/java/org/dspace/curate/DashStats.java
Though it’s pretty specific to the existing Dryad setup. It writes out a text file that is formatted for the counter-processor, but it’s sorted by dataset. Then there is a script that re-sorts everything based on time: https://github.com/datadryad/dryad-utils/blob/master/dash-migration/sort_dash_stats.sh

Happy to set up time for you to talk with Ryan Scherle (Dryad) if that would be helpful. Otherwise, the DataCite and DataONE folks may also have some tips.

[djbrooke]

Thanks @dlowenberg! I'll check in with the team here and we'll get back with you if we feel a discussion with Dryad is needed. Thanks again!

P.S. I just pinged you on another issue in the main Dataverse repo: IQSS/dataverse#5957

[djbrooke]

• We should do the things outlined in the original comment and other items not yet identified
• The current suggestion is to seed the count with the downloads that already exist, but we can discuss during the sprint.
• We should make note for users about how the numbers are derived (some from before the standard was implemented and others from after)

[djbrooke]

I picked this up out of the sprint column today to begin stubbing out documentation regarding migrating counts and other things that installations will need to know to use Make Data Count in production, but I don't have the bandwidth this week. I will re-visit early next week.

[pdurbin]

@djbrooke if you're stubbing out documentation, you might want to create a branch for IQSS/dataverse#6082 which was just opened. The issue title is "Documentation: Some tweaks to Make Data Count doc based on recent experience".

[jggautier]

See #75 (comment)

[cmbz]

• I moved this issue into the Global Backlog in the NIH Backlog column, as per conversations with @siacus and current AIM 5 Year 2 plans.

[landreev]

One random thought: one of the prod. servers, dvn-cloud-rserv-1.lib.harvard.edu is currently underutilized, and could be a prime candidate for running that processor on the accumulated logs.

[pdurbin]

Heads up that Counter Processor was archived yesterday:

• information about archiving the repository. CDLUC3/counter-processor#32

[pdurbin]

As discussed at standup, I forked the repo:

• Release Counter Processor under gdcc, switch docs from CDLUC3 dataverse#10406

[sbarbosadataverse]

• Monitor: Status of Make Data Count ingest metrics dataverse-pm#218

[pdurbin]

This was accidentally and automatically closed when IQSS/dataverse#10424 was merged. Re-opening.