Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DC2 - T2.2 - CKAN Data Collection #25

Closed
ghilbrae opened this issue Jan 22, 2020 · 18 comments
Closed

DC2 - T2.2 - CKAN Data Collection #25

ghilbrae opened this issue Jan 22, 2020 · 18 comments
Assignees
@LenaStr
Labels
data help wanted Extra attention is needed validation validation realted issues
Milestone
D7.10 Data Manage...

Comments

@ghilbrae
Copy link

We've tested the information provided by you in CKAN with special interest on the URLs provided for the Resources in the Datasets. The results can be seen in the attached file with the following structure:

DATASET: {dataset_name}

RESOURCE: {resource_name} (if there's a name)

URL {ERROR | EXISTS} -> {url} (if there's one)

dc2.txt
@ghilbrae ghilbrae added data help wanted Extra attention is needed labels Jan 22, 2020
@ghilbrae
Copy link
Author

In the attached sheet I've made an analysis of the data available in CKAN and added some comments on its status.
Please take a look at it and address the comments.
If there's data no longer in use, please delete the datasets. Same for datasets that might be in use that are not here yet.

Also, if you have changed anything in the last couple of days, those changes may not be reflected there yet, I've been working on verifying the data for a few days now. Please, let me know and update the comments anyway.

Status of DS+Resources available in CKAN_DC2.xlsx

@p-a-s-c-a-l p-a-s-c-a-l added the validation validation realted issues label Mar 30, 2020
@p-a-s-c-a-l
Copy link
Member

The following datasets in CKAN are missing the mandatory assignment to one of the 4 groups:

"Dataset Title" "organisation" "url"
"Heat scenarios over Stockholm" "DC2 - Sweden" "https://ckan.myclimateservice.eu/dataset/heat-scenarios-over-stockholm"

@LenaStr Please update datasets.

@p-a-s-c-a-l p-a-s-c-a-l mentioned this issue Apr 1, 2020
Closed
9 tasks
@LenaStr
Copy link

LenaStr commented Apr 1, 2020 via email

@p-a-s-c-a-l
Copy link
Member

Status of DS+Resources available in CKAN_DC2.xlsx

FYI: From data management perspective it is o.k. if both used and produced non-open data is stored on internal CLARITY sFTP.

@p-a-s-c-a-l
Copy link
Member

Pascal, it seems that I can not access CKAn for the moment due to security issues. Do you know what is wrong?

I've installed some security updated. Please try again.

@LenaStr
Copy link

LenaStr commented Apr 1, 2020 via email

@p-a-s-c-a-l
Copy link
Member

p-a-s-c-a-l commented Apr 3, 2020 •
edited
Loading

Please have a look at the Data Management Validation Report and correct all errors.

@LenaStr
Copy link

LenaStr commented Apr 3, 2020 via email

@p-a-s-c-a-l
Copy link
Member

I have gone through them and believe I solved most of the issues.

thanks!

What to do with datasets not available online, I have no link to provide there?

In that case I would just state that in the description of the dataset and omit the link.

@LenaStr
Copy link

LenaStr commented Apr 6, 2020 via email

@p-a-s-c-a-l
Copy link
Member

p-a-s-c-a-l commented Apr 23, 2020 •
edited
Loading

The current list of exploitable results contains the following data packages offered by SHMI:

  • Hydrological hazard resource for EU
  • Future Hydrological resource for Sweden
  • Local heat hazard resources for Stockholm

The license is CC Attribution 4.0 International (CC BY 4.0)

Therefore this data must be reported in the DMP as open-data produced by CLARITY. ATM there is just one dataset (Future flooding in Sweden) in the group Open Data produced by CLARITY.

@LenaStr
Copy link

LenaStr commented Apr 24, 2020

Then let me know how it should be:

Hydrological hazard resource for EU - this one is freely available (dont know if anyone has integrated it into CSIS). Produced in a projecc prior to Clarity.

Future Hydrological resource for Sweden - this one will be available on request. (Conditions may vary dependent on use, so maybe bettar to put restricted here).
Local heat hazard resources for Stockholm - This is a collection of data produced in other projects and in improvd in Clarity. Freely available.

/Lena

@p-a-s-c-a-l
Copy link
Member

O.K. Thanks. This was also stated in the previous DMP:

DC2 produces two datasets that are of general interest to release as open data: The scenarios for flooding in Sweden under a future climate and the urban heat scenarios over Stockholm. SHMI, as one of the data providers, intends to release these data as open by the end of the project. However, due to new regulations in Sweden SHMI might be restricted in what they are allowed to release. This issue will be clarified until the release of the next version of the DMP (deliverable D7.10).

So we should concentrate on the Future Hydrological resource for Sweden and Local heat hazard resources for Stockholm since those are datasets originally produced within CLARITY. Those datasets are already describe din CKAN:

Future flooding in Sweden is declared as non open. This is o.k. as long as we can give a reason (privacy, commercial exploitation interest, input data is not open, etc.). So if you could make a short statement on this in the resource description, this should be sufficient for the DMP. However, we are required to make a statement on how long term availability of this data is guaranteed, even if it's not openly available. That's open for discussion here.

Heat scenarios over Stockholm is declared as open-data and download links pointing to urban-sis.smhi.se are available. Now the question is if this qualifies as long-term archival as requested by data management requirements. Here we have two possibilities:

  1. either zip the data of and put them on Zenodo (I can take care of this), or
  2. make a clear statement in the DMP document that the data hosted on urban-sis.smhi.se will be accessible at least 5 years after the end of the project.

I'm in favour of option 1) since this will link the dataset as open result with CLARITY project in OpenAire and the participant portal.

@p-a-s-c-a-l
Copy link
Member

BTW, another advantage of uploading Heat scenarios over Stockholm to Zenodo is that is automatically linked to CLARITY results in CORDIS.

@p-a-s-c-a-l
Copy link
Member

Heat scenarios over Stockholm is declared as open-data and download links pointing to urban-sis.smhi.se are available. Now the question is if this qualifies as long-term archival as requested by data management requirements. Here we have two possibilities:

1. either zip the data of and put them on Zenodo (I can take care of this), or

2. make a clear statement in the DMP document that the data hosted on urban-sis.smhi.se will be accessible at  least 5 years after the end of the project.

I'm in favour of option 1) since this will link the dataset as open result with CLARITY project in OpenAire and the participant portal.

@LenaStr Any comment regarding this issue?

@p-a-s-c-a-l
Copy link
Member

Dataset 'Heat scenarios over Stockholm' is now available on Zenodo.

@p-a-s-c-a-l
Copy link
Member

p-a-s-c-a-l commented May 7, 2020 •
edited
Loading

There are only 2 DC2 output datasets in CKAN but in the DMP we talk about three. Meta-Data on Detailed flooding data over Stockholm is missing in CKAN.

@LenaStr
Copy link

LenaStr commented May 8, 2020 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@LenaStr LenaStr

Labels
data help wanted Extra attention is needed validation validation realted issues
Projects
None yet
Milestone
D7.10 Data Management Plan v3
Development

No branches or pull requests
3 participants
@ghilbrae @p-a-s-c-a-l @LenaStr