Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature request: create a harvested dataset via the API #7330

Open
hakbailey opened this issue Oct 14, 2020 · 8 comments
Open

Feature request: create a harvested dataset via the API #7330

hakbailey opened this issue Oct 14, 2020 · 8 comments
Labels

Comments

@hakbailey
Copy link

Some background: MIT is working on a research data registry using Dataverse as the repository, and as part of this project we want to ingest metadata into Dataverse such that it is indexed and searchable but the search result points to the source repository instead of a local Dataverse record. This is the exact functionality of harvested records, however we are building our own harvesting service because we need to perform actions not possible in the internal harvester, like parsing returned records to identify MIT-affiliated authors and enhancing the metadata with data from our local data warehouse.

In talking to the Dataverse tech team about this, the question rose of whether it would be feasible/useful to add the ability to create "harvested" datasets via the API. They suggested we add it here as an issue to prompt discussion...questions, comments, and suggestions welcome!

@poikilotherm
Copy link
Contributor

poikilotherm commented Oct 20, 2020

This might be interesting for us along the way, as we have a similar idea for Jülich DATA, mostly targeted to being a metadata -based indexer of all scholarly research data dropping on and off campus.

We haven't decided yet if we would try to look into the https://github.com/IQSS/doi2pmh-server project, but this might be a benefit for that project, too. @RightInTwo wasn't your preferred way back then to push rather than pull?

@RightInTwo
Copy link
Contributor

Hi! I'm sorry to say that I forgot to follow up on that project and do not work in the field of research data anymore/for now. My apologies go out to @tcoupin because I did not give you an update about discontinuing my employment.

But maybe there is an interest to pick it up now, again. The general approach of providing a server that allows to make custom OAI-PMH sets from DOIs (which are not necessarily in a set at the source or even at the same source at all) seems to serve multiple use cases, see #5402 and https://github.com/IQSS/doi2pmh-server/blob/master/README.md.

Even though I'm currently not in a position to invest time, I'd be glad to take part in the discussion.

@djbrooke
Copy link
Contributor

Thanks @hakbailey for creating the issue. It'll be good to follow along with the work in #7325 from @qqmyers, as that may serve your use case as well.

@pacian
Copy link

pacian commented Jun 13, 2022

UCPH would be very interested in this long standing feature request. I think it would be well worth upstreaming this feature to the larger community if it should ever come to life.

@poikilotherm
Copy link
Contributor

UCPH would be very interested in this long standing feature request. I think it would be well worth upstreaming this feature to the larger community if it should ever come to life.

@pacian You might also be interested in a recent post of mine to the mailing list: https://groups.google.com/g/dataverse-community/c/L4HivEHETUI

@pdurbin pdurbin added Type: Feature a feature request Feature: API User Role: API User Makes use of APIs labels Oct 8, 2023
@pdurbin
Copy link
Member

pdurbin commented Oct 12, 2023

I'm just adding the word "import" here because I keep having trouble finding this issue. The way I keep thinking of this issue is something like like: I want to import a dataset and have it appear as if it had been harvested.

@DS-INRAE
Copy link
Member

@pacian You might also be interested in a recent post of mine to the mailing list: https://groups.google.com/g/dataverse-community/c/L4HivEHETUI

@poikilotherm has your fork been implemented on your installation since ? :)

@poikilotherm
Copy link
Contributor

Unfortunately, not yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: Important
Status: ⚠️ Needed/Important
Development

No branches or pull requests

8 participants