Skip to content

Commit

Permalink
doc tweaks for MDC processingState API #10424
Browse files Browse the repository at this point in the history
  • Loading branch information
pdurbin committed Mar 27, 2024
1 parent 82e35b8 commit 6cb9a4c
Show file tree
Hide file tree
Showing 2 changed files with 27 additions and 21 deletions.
14 changes: 4 additions & 10 deletions doc/release-notes/10424-new-api-for-mdc.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,5 @@
The API endpoint `api/admin/makeDataCount/{yearMonth}/processingState` has been added to Get, Create/Update(POST), and Delete a State for processing Make Data Count logged metrics
For Create/Update the 'state' is passed in through a query parameter.
Example
- `curl POST http://localhost:8080/api/admin/makeDataCount/2024-03/processingState?state=Skip`
(Please put at the bottom of the list under 🌐 API)

Valid values for state are [New, Done, Skip, Processing, and Failed]
'New' can be used to re-trigger the processing of the data for the year-month specified.
'Skip' will prevent the file from being processed.
'Processing' shows the state where the file is currently being processed.
'Failed' shows the state where the file has failed and will be re-processed in the next run. If you don't want the file to be re-processed set the state to 'Skip'.
'Done' is the state where the file has been successfully processed.
### Experimental Make Data Count processingState API

An experimental Make Data Count processingState API has been added. For now it has been documented in the developer guide: https://guides.dataverse.org/en/6.2/developers/make-data-count.html#processing-archived-logs
34 changes: 23 additions & 11 deletions doc/sphinx-guides/source/developers/make-data-count.rst
Original file line number Diff line number Diff line change
Expand Up @@ -88,21 +88,33 @@ To read more about the Make Data Count api, see https://github.com/datacite/sash

You can compare the MDC metrics display with the Dataverse installation's original by toggling the ``:DisplayMDCMetrics`` setting (true by default to display MDC metrics).

New Make Data Count Processing for Your Dataverse Installation
--------------------------------------------------------------
Processing Archived Logs
------------------------

A new script (release date TBD) will be available for processing archived Dataverse log files. Monthly logs that are zipped, TARed, and copied to an archive can be processed by this script running nightly or weekly.
The script will keep track of the state of each tar file they are processed. Through the following APIs the state of each file can be checked or modified.
Setting the state to 'Skip' will prevent the file from being processed if the developer needs to analyze the contents.
'Failed' files will be re-tried in a later run.
'Done' files are successful and will be ignored going forward.
The file(s) currently being processed will have the state 'Processing'.
The states are [NEW, DONE, SKIP, PROCESSING, FAILED]
The script will process the newest set of log files (merging files from multiple nodes) and calling counter_processor.
The Admin APIs to manage the states include a GET, POST, and DELETE(For Testing).
yearMonth must be in the format yyyymm or yyyymmdd

The script will keep track of the state of each tar file they are processed and will make use of the following "processingState" API endpoints, which allow the state of each file to be checked or modified.

The possible states are new, done, skip, processing, and failed.

Setting the state to "skip" will prevent the file from being processed if the developer needs to analyze the contents.

"failed" files will be re-tried in a later run.

"done" files are successful and will be ignored going forward.

The files currently being processed will have the state "processing".

The script will process the newest set of log files (merging files from multiple nodes) and call Counter Processor.

APIs to manage the states include GET, POST, and DELETE (for testing), as shown below.

Note: ``yearMonth`` must be in the format ``yyyymm`` or ``yyyymmdd``.

``curl -X GET http://localhost:8080/api/admin/{yearMonth}/processingState``

``curl -X POST http://localhost:8080/api/admin/{yearMonth}/processingState?state=done``

``curl -X DELETE http://localhost:8080/api/admin/{yearMonth}/processingState``

Resources
Expand Down

0 comments on commit 6cb9a4c

Please sign in to comment.