Metrics APIs: Dataverses, Datasets, Files

[djbrooke]

Following the discussion in #4478, we decided to add some new metrics endpoints to Dataverse based on the model that's current provided by Miniverse:

https://services.dataverse.harvard.edu/static/swagger-ui/index.html?url=/miniverse/metrics/v1/swagger.yaml

Let's estimate this as a group (all three endpoints) and if the estimate comes back high we can discuss breaking it up.

[djbrooke]

• Meeting notes: https://docs.google.com/document/d/1BlFUaZvCUkfqjx6Ahz66odTysRPyAAx7rfcY0X5Ny_w/edit

[pdurbin]

Today I stubbed out some code at fa8b601

[pdurbin]

In order to make this a little more real, I created a repo at https://github.com/IQSS/metrics.dataverse.org to start switching the front end code from pulling data from miniverse to pulling data from an installation of Dataverse. For now that installation of Dataverse is my server at https://dev1.dataverse.org running c62e7bb.

I started with what I suspect is the easiest endpoint to migrate, which is called "Dataverses by Category". Below you can see https://dataverse.org/metrics on the left and my implementation from the new repo above on the right. As I migrate the other five metrics over, they'll appear in the version on the right:

I guess I'll add a todo list. There are 6 metrics to implement:

• Dataverses Added Over Time. Started in 4e3c958
• Dataverses by Category. Done in fa8b601
• Datasets Added Over Time. Started in fdd609b
• Datasets by Subject. Started in 40ae4b1
• Files Added Over Time
• File Downloads Over Time. Started in 9352882 but needs work.

From the perspective of Swagger, these are the 6 from https://services.dataverse.harvard.edu/static/swagger-ui/index.html?url=/miniverse/metrics/v1/swagger.yaml

[djbrooke]

Discussed with @pdurbin (and got a demo!). Adding the six endpoints above to the Dataverse application is the correct scope of this issue. There are some other endpoints that are interesting (bytes used, for example), but we can develop iteratively. We'll work on an updated aggregator/reporting application in later issue, but we want to get these initial endpoints into a release sooner instead of later.

[pdurbin]

I just chatted with @dlmurphy and it sounds like he and @jggautier may have already written some of the remaining queries we need as part of #4169. I can, of course, also refer to the miniverse code and maybe even run miniverse on my laptop to have it show me the raw SQL queries. The miniverse code seems to use the Django ORM, which is an abstraction.

[pdurbin]

There were minor merge conflicts (methods added in the same place in Admin.java) so I just resolved them in 6cdad11

[kcondon]

OK, generally works well but found a few items:

• Might want to mention the default cache timeout in the docs.
• Many of the query endpoints use concat() which is incompatible with Postgres 8.4
• Datasets by subjects counts seem correct in small test env but way off on prod db (9k ui versus 22k api)
• datasetsBySubject and dataversesByCategory do not update their lastcalleddate value when accessed and do not seem to clear value if cache times out.

[matthew-a-dunlap]

Caching issue should be fixed with latest commit

[djbrooke]

(sorry for the delay in this comment)

@kcondon - @matthew-a-dunlap volunteered to take on the remaining issues in standup. I left it assigned to you in case you wanted to test the fixes in parallel, but moving it back to development is OK too.

[matthew-a-dunlap]

While fixing the issues found during QA, a few extra issues were found and resolved:

• The datasetsBySubject query no longer has a date component. The new query originally had a check on the month.
• The 'byMonth' queries were renamed to be 'toMonth', as this is more descriptive of their cumulative nature.
  • The metrics api documentation better describes this as well

All but one of the queries were touched during these changes, and they all should probably be re-tested.