3648 Change "DVN" to default value "producer" and only write the firs… #7094

JingMa87 · 2020-07-16T18:53:26Z

…t geoBndBox element with four geo coordinates.

What this PR does / why we need it: Improves the DDI XML according to the XSD

Which issue(s) this PR closes: 3648 (doesn't close it, but it's a partial fix)

Closes nothing yet, it's an improvement

Special notes for your reviewer: It's an improvement in the DDI XML response, but more is needed in the future.

Suggestions on how to test this: Add multiple Geographic Bounding Boxes to your dataset, one with only 3 coordinates, and two with four coordinates. Only the first box with four coordinates should be in the DDI XML.

Does this PR introduce a user interface change? If mockups are available, please link/include them here: No

Is there a release notes update needed for this change?: Yes

Additional documentation:

…ate the external "custom download" service. Everything else is done by an outside standalone program (a java program with its own pom file). (#6505)

still working on the documentation, so will need to check it in later.

added some info to the documentation explaining how the zipper does its thing. (#6505)

…I as well.(#6505)

…ecutable side. (#6505)

(fixed merge conflicts w/develop - mostly the POST handling added for the /api/access/datafiles/ API)

…section (#6505)

Co-authored-by: Philip Durbin <philip_durbin@harvard.edu>

…tom/service/download/ZipDownloadService.java Co-authored-by: Philip Durbin <philip_durbin@harvard.edu>

…/dataverse into 6505-optimize-zip-downloads

…uction

…ced guide; #6505

Co-authored-by: Philip Durbin <philip_durbin@harvard.edu>

mheppler · 2020-07-20T15:24:44Z

@jggautier if you believe that issue should remain open, you can edit the PR description to remove the "Closes" keyword.

I also added a comment references this PR to other older DDI XML issues I found in the backlog.

Add "download all" buttons (including size of dataset) to dataset page

Public ORCID login is available.

make ec2-create-instance.sh downloadable #7099

JingMa87 · 2020-07-20T22:06:44Z

@jggautier if you believe that issue should remain open, you can edit the PR description to remove the "Closes" keyword.

I also added a comment references this PR to other older DDI XML issues I found in the backlog.

I removed the Closes keyword and will take a look at the issues you referenced.

pdurbin

I'm sending this to QA because I think there is incremental improvement in this pull request but here are a few additional thoughts:

This pull request might be better as two smaller ones. One would remove "DVN" from "source". The other would change the behavior of the bounding box for geospatial data. (There could be separate issues for these as well.)
It would be good to test the DDI before and after with a tool that does XML schema validation. I assume that there will be fewer failures in this pull request compared to develop. I haven't done this myself and I'm relying on what @JingMa87 has coded up and comments by @jggautier in terms of the geospatial changes being correct. It probably makes sense to someday put the XML schema validation into a test so we can continue to get it passing and keep it passing.
This pull request changes "source" from "DVN" to "producer" but it seems like "archive" is another option. I don't feel strongly about this but some DDI experts that are part of the Dataverse community might.

pdurbin · 2020-07-21T18:40:44Z

src/main/java/edu/harvard/iq/dataverse/export/ddi/DdiExportUtil.java

@@ -328,7 +328,7 @@ private static void writeDocDescElement (XMLStreamWriter xmlw, DatasetDTO datase

    private static void writeVersionStatement(XMLStreamWriter xmlw, DatasetVersionDTO datasetVersionDTO) throws XMLStreamException{
        xmlw.writeStartElement("verStmt");
-        writeAttribute(xmlw,"source","DVN"); 
+        writeAttribute(xmlw,"source","producer"); 


The string "DVN_3_0" appears in if (!SOURCE_DVN_3_0.equals(xmlr.getAttributeValue(null, "source"))) { in ImportDDIServiceBean. It's not an exact match so I don't think this will break dataset import from DDI but I thought I'd at least mention it.

To me it seems like the "DVN_3_0" is a legacy value and will only still be used if an old Dataverse instance hasn't upgraded yet.

djbrooke · 2020-07-21T18:52:57Z

@JingMa87 @jggautier before we move this to QA, do you have any thoughts about @pdurbin's comments?

pdurbin · 2020-07-21T20:29:21Z

One last think to mention is that I brought up in tech hours today the idea of changing "source" from "DVN" to something more reasonable. No one objected and it sounds like it's a well known problem that "DVN" doesn't validate.

jggautier · 2020-07-21T20:47:03Z

I agree about creating a new issue for the bounding box changes. Hopefully the <verStmt> change is easy to make. The bounding box change needs some more discussion I think. @JingMa87, I can create a new issue for the bounding box change if you'd like.

So this PR can be the PR for only the <verStmt> change, and changes related to the bounding boxes can be removed?

I would use "archive" instead of "producer," mainly because that's what's used for the "distrbtr source". But also in other places in the DDI, when source = "producer", that means the information is coming from the depositor(s) and when source = "archive", that means the information is coming from the repository. The metadata document version number is automatically generated by the repository, so maybe that's another argument for using "archive".

So right now the version statement for the metadata document looks like this:

<verStmt source="DVN">
    <version date="2020-07-21" type="RELEASED">1</version>
</verStmt>

"DVN" isn't a valid value for "source". So @JingMa87 proposed changing it to producer:

<verStmt source="producer">
    <version date="2020-07-21" type="RELEASED">1</version>
</verStmt>

Again, the distribution statement already uses "archive":

<distStmt>
    <distrbtr source="archive">QDR Main Collection</distrbtr>
    <distDate>2020-07-06</distDate>
</distStmt>

So I think the version statement should also use "archive":

<verStmt source="archive">
    <version date="2020-07-21" type="RELEASED">1</version>
</verStmt>

What's lost in this change is the "DVN". In the current DDI export, I'm not sure what's being expressed by <verStmt source="DVN">. Is it that the source of the version statement is the Dataverse software itself, as opposed to the repository? Is this a holdover from when there was just one "Dataverse Network"? I don't think this information is being used for anything and it's fine that it's lost in the change from this PR.

JingMa87 · 2020-07-22T03:48:18Z

I also agree about separating the issues. @jggautier I already made an issue for the geoBndBox if you don't mind: Should the Geographic Bounding Box allow optional coordinates and multiple boxes in the XML? #7108
I tried validating the XML with the codebook, but it's not working and is giving me many pre-validation errors. @jggautier Is it still working for you?
I changed the value to archive and reverted the geoBndBox changes.

…t geoBndBox element with four geo coordinates.

…ix-ddi-export # Conflicts: # src/main/java/edu/harvard/iq/dataverse/export/ddi/DdiExportUtil.java # src/test/java/edu/harvard/iq/dataverse/export/ddi/dataset-finch1.xml # src/test/java/edu/harvard/iq/dataverse/export/ddi/exportfull.xml

JingMa87 · 2020-07-22T05:08:05Z

@pdurbin How come other people's commits are in this PR? Did I use a wrong command?

jggautier · 2020-07-22T18:43:45Z

Thanks @JingMa87. About validating the XML with the codebook and pre-validation errors, when I test a codebook XML export of a dataset that has only the metadata required fields, the only other error I get (besides the "DVN" error) is about the use of <setAvail/> <useStmt/>. Is that what you meant by pre-validation errors?

pdurbin · 2020-07-22T20:49:21Z

@JingMa87 I don't know. Git is hard. 😄 I'm pretty quick to make new branches at any sign of trouble. And it looks like both @jggautier and I would prefer the "source" change to be made in a dedicated branch/pull request.

JingMa87 · 2020-07-24T13:34:29Z

@jggautier I actually meant that the imports that the XSD uses were faulty and prevented the validation from happening. But I fixed these in the meantime and all is well. Maybe I'll incorporate an XSD validation in the source code sometime in the future.
@pdurbin I made a separate issue for the DVN problem #7128 and will submit a PR soon. While validating the XML I came across another problem, namely the order of some of the elements. This is easy to fix and the issue can be found here #7127. I'll do it in the near future.

landreev and others added 30 commits May 29, 2020 10:50

The (very limited) changes that went into the application to accommod…

b8d268a

…ate the external "custom download" service. Everything else is done by an outside standalone program (a java program with its own pom file). (#6505)

components of the standalone zipper (#6505).

8b1765a

still working on the documentation, so will need to check it in later.

handling of folders added to the zipper;

e3973d1

added some info to the documentation explaining how the zipper does its thing. (#6505)

cosmetic (#6505)

ad1787a

The modifications allowing the use of the "custom zipper" with the AP…

1dc597b

…I as well.(#6505)

uncommented the line that cleans the request table, on the service ex…

ddfc88c

…ecutable side. (#6505)

Merge branch 'develop' into 6505-optimize-zip-downloads

5402e07

(fixed merge conflicts w/develop - mostly the POST handling added for the /api/access/datafiles/ API)

a release note for the "zipper tool". (#6505)

aa923ba

added a section on the zipper service to the "installation/advanced" …

5d27982

…section (#6505)

adding new setting to release notes

c99fa60

Public ORCID login is available.

c10b516

Update scripts/zipdownload/README.md

8dfe4c4

Co-authored-by: Philip Durbin <philip_durbin@harvard.edu>

Update scripts/zipdownload/src/main/java/edu/harvard/iq/dataverse/cus…

46584da

…tom/service/download/ZipDownloadService.java Co-authored-by: Philip Durbin <philip_durbin@harvard.edu>

Better/safer handling of database queries (#6505)

5aaaff5

Merge branch '6505-optimize-zip-downloads' of https://github.com/IQSS…

48a56df

…/dataverse into 6505-optimize-zip-downloads

added a line about the Apache configuration to the installation instr…

3eb3976

…uction

line breaks in the readme (#6505)

1cd8629

small addition to the guide on installation (#6505)

69297fb

documents the zipper setting. (#6505)

6e2e396

fixes "original" always being true (#6505)

72394a4

removed unnecessary repos from pom.xml; a few more words in the advan…

96c3708

…ced guide; #6505

Update doc/sphinx-guides/source/installation/advanced.rst

e01c213

Co-authored-by: Philip Durbin <philip_durbin@harvard.edu>

Update scripts/zipdownload/README.md

9e42aec

Co-authored-by: Philip Durbin <philip_durbin@harvard.edu>

style/grammar #6505

6100ed6

Update scripts/zipdownload/README.md

aaaa035

Co-authored-by: Philip Durbin <philip_durbin@harvard.edu>

Update scripts/zipdownload/README.md

c5cca50

Co-authored-by: Philip Durbin <philip_durbin@harvard.edu>

Update scripts/zipdownload/README.md

1d4b83f

Co-authored-by: Philip Durbin <philip_durbin@harvard.edu>

typo

d34ecca

Fix for NPE in test.

55c2c89

Oauth documentation is extended with ORCID public API option.

8da201b

mheppler mentioned this pull request Jul 20, 2020

Adjust how new variable metadata is added to DDI XML exports so that the exports are valid against DDI schema #6554

Closed

kcondon added 3 commits July 20, 2020 14:58

Merge pull request #7047 from IQSS/6118-download

d3f1f6d

Add "download all" buttons (including size of dataset) to dataset page

Merge pull request #7025 from dsd-sztaki-hu/orcid-public-api

f5104d1

Public ORCID login is available.

Merge pull request #7100 from IQSS/7099-docs

941d17d

make ec2-create-instance.sh downloadable #7099

pdurbin approved these changes Jul 21, 2020

View reviewed changes

djbrooke added this to the Dataverse 5 milestone Jul 21, 2020

djbrooke assigned JingMa87 and jggautier Jul 21, 2020

djbrooke removed this from the Dataverse 5 milestone Jul 21, 2020

JingMa87 added 3 commits July 22, 2020 06:11

3648 Change "DVN" to default value "producer" and only write the firs…

fe3b901

…t geoBndBox element with four geo coordinates.

3648 Change value to archive.

6942c90

JingMa87 closed this Jul 24, 2020

JingMa87 deleted the 3648-fix-ddi-export branch July 24, 2020 20:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

3648 Change "DVN" to default value "producer" and only write the firs… #7094

3648 Change "DVN" to default value "producer" and only write the firs… #7094

JingMa87 commented Jul 16, 2020 •

edited

Loading

mheppler commented Jul 20, 2020

JingMa87 commented Jul 20, 2020

pdurbin left a comment

pdurbin Jul 21, 2020

JingMa87 Jul 22, 2020

djbrooke commented Jul 21, 2020

pdurbin commented Jul 21, 2020

jggautier commented Jul 21, 2020 •

edited

Loading

JingMa87 commented Jul 22, 2020 •

edited

Loading

JingMa87 commented Jul 22, 2020

jggautier commented Jul 22, 2020 •

edited

Loading

pdurbin commented Jul 22, 2020

JingMa87 commented Jul 24, 2020 •

edited

Loading

3648 Change "DVN" to default value "producer" and only write the firs… #7094

3648 Change "DVN" to default value "producer" and only write the firs… #7094

Conversation

JingMa87 commented Jul 16, 2020 • edited Loading

mheppler commented Jul 20, 2020

JingMa87 commented Jul 20, 2020

pdurbin left a comment

Choose a reason for hiding this comment

pdurbin Jul 21, 2020

Choose a reason for hiding this comment

JingMa87 Jul 22, 2020

Choose a reason for hiding this comment

djbrooke commented Jul 21, 2020

pdurbin commented Jul 21, 2020

jggautier commented Jul 21, 2020 • edited Loading

JingMa87 commented Jul 22, 2020 • edited Loading

JingMa87 commented Jul 22, 2020

jggautier commented Jul 22, 2020 • edited Loading

pdurbin commented Jul 22, 2020

JingMa87 commented Jul 24, 2020 • edited Loading

JingMa87 commented Jul 16, 2020 •

edited

Loading

jggautier commented Jul 21, 2020 •

edited

Loading

JingMa87 commented Jul 22, 2020 •

edited

Loading

jggautier commented Jul 22, 2020 •

edited

Loading

JingMa87 commented Jul 24, 2020 •

edited

Loading