Making Data Count statistics

• COUNTER stats in Dash UI
  • Querying Crossref DataCite and Crossref events for citations, deduplicating and counting
  • Querying DataCite Event Data APIs for Views and Counts
  • Removed old stats view/download tables and ajax requests
  • UI Library icons/css
  • Table for caching and showing stats (updates once a day)
  • Unit tests with doubles/mocks for views/downloads/citations
• Counter-Proessor
  • Updated documentation. https://github.com/CDLUC3/Make-Data-Count/blob/master/getting-started.md
  • Optimizations (Sqlite Vacuum)
  • Set up Bash script for nightly run
  • Scheduled script

Misc Updates, Reduce Exception Spam

Since last release notes:

• Updated Merritt-Manifest code
• Added to OpenAPI (Swagger) docs for API
• Added documentation for write-access to API
• Modified settings for email exception notifier so it uses the option to only notify on 1, 2, 4, 8, 16, 32 repeats of an individual error within a time period (3 hours) since we had vulnerability scans setting off many thousands of emails
• Some work on retrieving MDC/Counter stats that hasn't been incorporated into the UI yet
• A lot of changes made to COUNTER processing code (though in a separate repository) that then will eventually feed back into the Dash system as we retrieve the stats we sent to DataCIte

Status update dashAPI, Counter, NOT DEPLOYABLE use previous or next tag

This is a progress status update and the full suite of engines are not deployable for this version.

Documentation

• How to add application and associated user (confluence operations)
• Examples for testing basics of Dash submission API https://github.com/CDLUC3/stash/blob/development/stash_api/basic_submission.md
• Updates to OpenAPI docs files, submission, editing DS, https://dash-dev.ucop.edu/api/docs/index.html

Dash API

• Refactored permissions so I can reuse user permissions in the API more easily
• PUT file to a resource
  • Many prerequisites:
    • authorized
    • in-progress
    • folder creation
    • disallow uploading outside of our root directory (with ../.. or other trickery)
    • size warnings if sent in headers
  • Accept upload payload and save to file system on server
  • Save file information to database
  • Check for size violations or other errors after upload
  • Convert file PUT to POST (because of named resource)
• PATCH to a dataset with JSON to publish, http://williamdurand.fr/2014/02/14/please-do-not-patch-like-an-idiot/
  • Using data validations before allowing submission, returns errors
  • Added missing default values like in the UI (type is dataset, publisher is campus/tenant, etc) because of errors
• PUT to dataset URL for replacing the metadata with different metadata for in-progress
  • Updates title, authors, abstract, keeps ORCIDs if set already in DB for dataset

Misc

• Fixes to mysql2 library incompatibilities between different gems/engines

Counter

• Clarified how DataCite wants us to send updates, put in tickets, we'll need revisions to request we send

Counter CoP, OAuth2 authorization, submission API changes

Counter (related to Dash)

• Added Counter processor changes at https://github.com/CDLUC3/counter-processor for submitting to sashimi hub
  • Added sample data for processing a report
  • Made Documentation changes
  • Made successful submission with a live version of the hub

Dash

• integrated doorkeeper oauth2 server into our API engine inside Dash
  • Worked around many problems of installing in an isolated engine
  • Enabled assigning ids/secrets by a superuser for an application (still need to do manual work to associate the user to this application, however)
  • Configured and enabled client credentials grant type
  • Token may be obtained from OAuth2 at /oauth/token
• Added test request testing OAuth bearer key authentication at /api/test
• Created POST request for adding basic metadata and getting a DOI for identifier
  • Added different flow for DOI generation which previously has happened at time of publication, this happens at time of metadata submission for a new dataset
  • Fixed bug around author affiliations being unreliable in the model
  • Added code to clean up empty values in JSON output
• Documented new requests in Swagger at api/docs/index.html
• Updated rubocop to same version across all engines
• Updated loofah and html sanitizer to mitigate security vulnerabilities

COUNTER Logging, file sizes from Merritt, UCSF Logo

• Get file size from Merritt if it never was returned by server
• Changes to log updated log information for COUNTER
  • logging descriptive metadata
  • adding header lines describing logging fields
  • simplified logger in UI code
  • API is logging dataset investigations and requests
• Changed download links in API to go to Dash URLs redirected to Merritt Repository URLs
• Changed UCSF logo
• Rake task to fill missing file sizes (from Merritt) before deploying

Place search changes, counter logging, counter processing, bug fixes

• Fixing tests for DataCite minting/submission
• Leaflet geosearch integration instead of mapzen
  • Adding Google map lookup (requires keys and other things from google)
• Counter logging
  • Normal and PDF investigations
  • Many "requests": Embargoed & owner, secret link, public, large public downloads (emailed from Merritt)
• Increasing Merritt timeouts for size callbacks we were getting some failures
• Adding tenant to resource (identifier) so we don't rely on user accounts
  • Code changes
  • Fixing many broken tests
• Adding processing for our logs to deduplicate clicks and get geoip

Read-only initial version of API

This release includes some OpenAPI documentation and an initial/preliminary version of a GET, read-only API for feedback.

Features:

• GET datasets /api/datasets
• GET a dataset /api/datasets/
• GET the versions of a dataset /api/datasets//versions
• GET a version specific version /api/versions/
• GET a file information about a version /api/versions//files
• GET file information /api/files/
• Download an individual file ( /api/downloads/ ) or download other packages as shown in links
• See API documentation at /api/docs/index.html
• It includes HAL links for discovering related resoures

Individual file downloads, data deposit agreements, security enhancements and bug fixes

• Adding integration testing changes to get CKEditor forms working with integration tests
• Adding data deposit agreement for UC Davis
• Individual file downloads
  • Downloading individual files from Merritt Repository if public
  • When logged in, can download delayed release for your own dataset (streaming through our UI)
• Consistent model for user/permission validation for adding/changing/deleting data by AJAX
  • All forms/fields on metadata entry
  • File Uploads use common security
  • Embargo setting on review page uses same security methods
• Disallow font-face changes when people copy & paste

Rich text editing, email changes, bug fixes

• Integrated CKEditor for use with description fields of Abstract, Methods and Usage Notes.

  • Configured widgets available
  • Integrated Javascript and CSS
  • Enabled AJAX auto-submission when exiting
  • Removing HTML tags on submission to DataCite since they do not support HTML
  • Removing script tags or unsafe tags from entry.

• Created data migration script to move old plain text into a reasonable HTML format to use for continued editing going forward (includes our autolinking and paragraphing)

• Model changes to enable easy manual dataset deletion & deleted two rogue datasets

• Fixing manifest bug to set mime-type to 'application/octet-stream' if not set

• Adding campus contacts that are blind copied by email on submissions

• Fixing Javascript affiliation bug that appeared only when multiple users added in a different order

• Adding "Dash" into publisher

• Changing contact email address for UCI

• Experimented with substance editor and substance forms, wasn't suitable for our time frame and features that were requested

  • Got features/css working with substance-forms editor (in separate branch)

minor tweaks to mime_type and other small fixes

• Configuring UC Press
• Troubleshooting missing mimetype error on manifest submission
• Cascading deletes to make it easier to delete records
• Rationalizing email addresses and use from config instead of the hard coded thing for exception notifications
• Fixed extra errors from Rubocop upgrade for Rails 5 items (locked it to our version)
• Removing auto-population of author affiliation (based on login)
• Configuring login and other settings and adding disclaimer for UC Press
• Adding Rinku gem that is supposed to autolink text URLs in a better way
• Adding size to admin table and getting from Merritt on successful submission
• Catching timeout from Merritt when we request the size
• Correct errors caused by routes in submission emails that were making application not work on stage/demo/prod
• UC Press & DataOne both use Google login. Possible problems if they overlap (added ticket).