DC4 - T2.2 - CKAN Data Collection

[ghilbrae]

We've tested the information provided by you in CKAN with special interest on the URLs provided for the Resources in the Datasets. The results can be seen in the attached file with the following structure:

DATASET: {dataset_name}

RESOURCE: {resource_name} (if there's a name)

URL {ERROR | EXISTS} -> {url} (if there's one)

dc4.txt

[LauraMTG]

Hi, Angela.

I have reviewed the dataset URLs and corrected them with current resources. However, not all of them have been

DATASET: meteorological-observation-data-hourly
RESOURCE:
ERROR URL -> Missing URL

DATASET: vegetation-condition-on-ditches-and-median-strips
RESOURCE:
ERROR URL -> Missing URL

DATASET: meteorological-observation-data-daily
RESOURCE:
ERROR URL -> Missing URL
DATASET: cmip5-climate-projections
RESOURCE:
ERROR URL -> Missing URL
DATASET: current-climate-atlas
RESOURCE:
ERROR URL -> Missing URL
DATASET: decadal-models-outputs-cmip5
RESOURCE:
ERROR URL -> Missing URL

I don't know the URL for these cases.

Translated with www.DeepL.com/Translator (free version)

[ghilbrae]

In the attached sheet I've made an analysis of the data available in CKAN.
Please take a look at it and address any issues.

If there's data no longer in use, please delete the datasets. Same for datasets that might be in use that are not here yet.

Also, if you have changed anything in the last couple of days, those changes may not be reflected there yet, I've been working on verifying the data for a few days now. Please, let me know and update the comments anyway.

Status of DS+Resources available in CKAN_DC4.xlsx

[LauraMTG]

@ghilbrae

I'm reviewing your bug report and comments. I'm just making the following notes:

-DATASET: mid-term-meteorological-forecasting-noaa. Is there a better URL? One that points to the actual dataset? It doesn't exist now.
-DATASET: digital-elevation-model-data-over-europe-eu-dem. Missing URL. I think this resource is misplaced in DC4, but the metadata seems to belong to Robert Goler ....

[p-a-s-c-a-l]

The following datasets in CKAN are missing the mandatory assignment to one of the 4 groups:

"Dataset Title"	"organisation"	"url"
"Exposure elements"	"DC4 - Spain"	"https://ckan.myclimateservice.eu/dataset/exposure-elements"

@ghilbrae Please update datasets.

[p-a-s-c-a-l]

Status of DS+Resources available in CKAN_DC4.xlsx

FYI: From data management perspective it is o.k. if both used and produced non-open data is stored on internal CLARITY sFTP.

[ghilbrae]

The following datasets in CKAN are missing the mandatory assignment to one of the 4 groups:
"Dataset Title" "organisation" "url"
"Exposure elements" "DC4 - Spain" "https://ckan.myclimateservice.eu/dataset/exposure-elements"

@ghilbrae Please update datasets.

This datasets are not ready yet. I'm waiting for them to be to update everything.

Status of DS+Resources available in CKAN_DC4.xlsx

FYI: From data management perspective it is o.k. if both used and produced non-open data is stored on internal CLARITY sFTP.

That is news for me. I remember being said that the FTP was temporary and would go away once the project was finished, so that made necessary to have some other place to store data, like CKAN, geoserver and/or ZENODO.

[p-a-s-c-a-l]

I remember being said that the FTP was temporary and would go away once the project was finished, so that made necessary to have some other place to store data, like CKAN, geoserver and/or ZENODO.

O.K. I didn't know that there are plans to shut down the FTP immediately after the project ends. Perhaps @maesbri can clarify this. At least the owner of the data should still have access to it and be able to provide non-open data on request. All open-data produced by the project has to be stored on Zenodo anyway.

Uploading non-open data to CKAN is not a good idea. You would have set the respective dataset to 'private' which makes the meta-data private too - not an option for the data management plan since all meta-data has to be public.

Best option would be to keep non-open data on the private sFTP.

[p-a-s-c-a-l]

Please have a look at the Data Management Validation Report and correct all errors.

[p-a-s-c-a-l]

Updated DC4 Validation Report: dc4.docx

There are still some issues, can you please correct them @LauraMTG ?

[p-a-s-c-a-l]

O.K. I didn't know that there are plans to shut down the FTP immediately after the project ends. Perhaps @maesbri can clarify this. At least the owner of the data should still have access to it and be able to provide non-open data on request. All open-data produced by the project has to be stored on Zenodo anyway.

Uploading non-open data to CKAN is not a good idea. You would have set the respective dataset to 'private' which makes the meta-data private too - not an option for the data management plan since all meta-data has to be public.

Best option would be to keep non-open data on the private sFTP.

We have to clarify this ASAP as we have to make a statement on long-term preservation of non-open data in the upcoming DMP!

[p-a-s-c-a-l]

BTW, there are no datasets produced by DC4 in CKAN neither open nor non-open. Is this really correct?

In the current list of exploitable results, the following data package is provided by AEMET as "open and free: Future climate hazard resources for Spain road networks.

Why are those datasets not in in CKAN and Zendo, respectively?

[p-a-s-c-a-l]

BTW, there are no datasets produced by DC4 in CKAN neither open nor non-open. Is this really correct?

@ghilbrae Any updates regarding this issue?

[ghilbrae]

The only datasets that will be produced are related to the FWI. We are going to upload them to Zenodo when they are ready for publishing. We will then add them to CKAN if necessary.

Also, Aemet is generating some data that will be stored in https://www.adaptecca.es/ when they ree ready we will reference them in CKAN too.

[p-a-s-c-a-l]

Also, Aemet is generating some data that will be stored in https://www.adaptecca.es/ when they ree ready we will reference them in CKAN too.

Can you please add the meta-data of those datasets now to ckan? We can upload the data later. We need the description of the data for the upcoming DMP, so no need to wait any longer.

[ghilbrae]

np, I'll get them from Aemet and my colleagues and add them.