Metadata resources

[fxprunayre]

A metadata can have a set of resources uploaded to the catalog. Those documents are stored in 2 folders in the data directory in data/metadata/{{folder-group-sequence}}/{{metadata_id}}/:

• public folder where no security checks are made to download the resources
• private folder where only users member of a group with download privileges can download the resources.

Uploaded resources can be files to link to the metadata record (eg. a ZIP containing the data in Shapefile, PDF) or overviews.

Motivations & current limitations

There is some limitations with current implementation which can cause trouble with uploaded files.

1. If an editor remove in the XML view (or editor view) a section with a reference to an uploaded document, the uploaded document may remain in the metadata directory without any references in the XML. This file can't be manage anymore and is lost because no UI provides access to this file.

2. There is no possiblity to set if the file is in the public or private folder to restrict its access.

3. Only one thumbnail can be uploaded (with the large and small version).

4. Only 2 types of related document can be associated to a record:

• thumbnail
• online resources (in the distribution section in ISO19139)
  Some other standards have the capability to reference document in many places (eg. in ISO19115-3, reference can be created in all CI_Citation - link to a feature catalogue, a data quality report, ...)

CFV 22 January 2016

• +1 Jeroen, Jose, Florent, Francois

Proposal

This proposal add a concept of a file store which takes care of managing resources related to a metadata record. It allows to:

• List metadata resources
• List public metadata resources
• List private metadata resources
• Upload a resource (by upload, by copy from URL)
• Download a resource
• Delete a resource
• Delete all resources

The file store provides URLs for all resources than can be used in the metadata document to link to a specific resource. Those URLs can be used for any kind of links in the metadata document (eg. gmd:linkage, gmx:Anchor/@href, gmd:graphicOverview).

Improvements:

• Editor can upload as many overviews as needed
• Overview can have multilingual description
• Overview can be public or private
• Dublin core record can have overview
• The first overview in a document is the "main" one - the one display in search results.
• Improve online source popup
  • Add configuration by standards
  • Add form validation
• Improve related resource panel
  • Add title for thumbnails and links
  • Improve thumbnail layout
  • Thumbnail can be open full screen (in record view and editor view)
• Data store files can always be managed even if no reference exist in the metadata document
• Editor can easily make a file public or private
• Editor can upload multiple files at once
• Restore thumbnail creation from WMS

Changes:

• New metadata resources API
• Use Swagger for API documentation
• Remove old services
  • Remove thumbnail & resource old services
  • Remove extract/set-thumbnails.xsl
• New dependencies
  • Add springfox http://springfox.github.io/springfox/docs/current/ for API documentation
  • Add swagger-ui for publishing API documentation

User interface

To link a document to a metadata record, choose the Add document menu from the related resources panel:

The add document panel is composed of 2 panels:

• on the left, describe the document to be added
• on the right, upload a new document or generate an overview

The file store panel allows to:

• upload one or more files
• change visibility of files (ie. private = user needs to have download privileges for the group, public)
• download a file
• open a file in new tab

To add a document, choose the type of document. For ISO19139, 2 types of document can be added:

• overview
• online source

This may be customized on a per schema basis using the configuration (eg. https://github.com/fxprunayre/core-geonetwork/blob/improvement/metadata/resources/web-ui/src/main/resources/catalog/components/edit/onlinesrc/OnlineSrcDirective.js#L155).

Once the document is uploaded, the editor is updated and the new document display in the list of resources:

API

This PR introduce Springfox and Swagger to generate API document. In the move to Spring MVC, it sounds relevant to improve the API documentation and make it easier to maintain at the code level.

Annotations are used to document the API eg. https://github.com/fxprunayre/core-geonetwork/blob/improvement/metadata/resources/services/src/main/java/org/fao/geonet/services/api/metadata/resources/ResourcesApi.java#L121.

Springfox take care of generating swagger API document cf. https://github.com/fxprunayre/core-geonetwork/blob/improvement/metadata/resources/services/src/main/java/org/fao/geonet/services/api/SwaggerConfig.java.

Then API can be easily published. Eg.

http://apps.titellus.net/gnmdr/swagger-ui/

Migration:

• Database migration update all links found in metadata records

Future improvements not part of that PR

• Create CatalogResources to add document to the catalog that can be referenced in all metadata records (eg. license)
• Store resource reference in database in order to be able to:
  • easily rename file
  • properly follow statistics on resources (related to Report module #436)
• Move schema configuration to schema plugins
• API
  • Security / Use @PreAuthorize annotation instead of intercept-url
  • Exceptions / localized message in Java code
  • Remove old API based on resources.get

[fgravin]

Very good work, looks promising !
+1

[landryb]

That looks very nice - just one thing, i dont see what triggers the database migration for urls ... i suppose it should be run only once ?

[landryb]

How compatible will this be with Lib.resource.getDir usage everywhere in the tree (https://github.com/geonetwork/core-geonetwork/search?utf8=%E2%9C%93&q=Lib.resource.getDir), which expects an access parameter, being sometimes hardcoded to private or public, or passed around via various API/url calls ?

[fxprunayre]

How compatible will this be with Lib.resource.getDir usage everywhere in the tree

The FileSystemStore using the same "private", "public" folder structure this will be the same. But at some point it could be more relevant to always use the store to access those resources.
BTW if we agree to start cleaning the code base (?), references from resources/, thumbnail/ will be dropped. Then the changes will apply mainly to MEF, Harvester.

[fxprunayre]

what triggers the database migration for urls

Added here https://github.com/geonetwork/core-geonetwork/pull/1385/files#diff-c8e221114d0ca80a1dba4bf25153acd3R137

[landryb]

Regarding private/public, if you can make sense of the workflow in the formatter cache at the same time.. the use of PRIVATE and PUBLIC in https://github.com/geonetwork/core-geonetwork/blob/3.0.x/services/src/main/java/org/fao/geonet/services/metadata/format/cache/FormatterCache.java is all voodoo to me.

[fxprunayre]

Regarding private/public, if you can make sense of the workflow in the formatter cache at the same time.. the use of PRIVATE and PUBLIC

With this PR the visibility of a resource is not anymore in the resource id so I don't think the formatter will be affected. URL remains the same.

is all voodoo to me.

:) for me too !

[aimsdc]

Hi,

Great improvement for managing attachments and thumbnails. Just some issues I had using other schema.

Schema config added/copied to OnlineSrcDirective.js didn't work on it's own. I also needed to define my schema in Geonetwork admin settings (metadata/editor/schemaConfig). If the schema is not defined here the widget is hidden on the edit page.

    <div class="col-md-4"
         data-ng-if="gnSchemaConfig.related.display === true || ... >
        <div data-gn-onlinesrc-list="" data-ng-if="gnSchemaConfig.related.display === true"></div>
    </div>

Came accross this in core-geonetwork\web-ui\src\main\resources\catalog\templates\editor\editor.html

<div data-gn-modal="" class="onlinesrc-popup" data-gn-popup-options="{title:'addThumbnailTitle'}" id="addthumbnail-popup">
  <div data-gn-add-thumbnail="" class="add-thumbnail" data-gn-popupid="#addthumbnail-popup"</div>
</div>

However I couldn't find the directive gnAddThumbnail, is this now obsolete?

The other issue i'm having is importing MEF2 files from older Geonetworks (2.10.3). Now we only have one image file (not previous _s version) and the complete URL to the image is stored in the XML gmd:graphicOverview . I can write some scripts to get around this, or even modify the old Geonetwork MEF2 export, could addressing this on the MEF2 import be a consideration for future improvements of this functionaity?

Thanks
Kevin (AIMS)

[fxprunayre]

However I couldn't find the directive gnAddThumbnail, is this now obsolete?

Yes. Dropped in 36999bf

could addressing this on the MEF2 import be a consideration for future improvements of this functionaity?

When you edit a record, the URL should be updated no (from filename to complete URL) ? Another option is maybe to run the migration task for this. See #1509