Request of addition of a new ISTAT provider

[elakkkk]

ISTAT is switching data platforms. As of now, the provider regUrl and repoUrl are "http://sdmx.istat.it/SDMXWS/rest" and are related to I.Stat platform (http://dati.istat.it/). The new platform is called IstatData (https://esploradati.istat.it/databrowser/) but there is no direct access with readSDMX yet, it works only through url. The request is adding a new provider, which is still ISTAT, but linked to the new platform, that has, other than the datasets of I.Stat (old platform), new datasets and new codelists. The idea is to maintain the old provider until ISTAT decides to dismiss I.Stat, while giving the possibility of accessing to the new platform. The new regUrl and repoUrl are "https://esploradati.istat.it/SDMXWS/rest/" (it can be checked at https://esploradati.istat.it/SDMXWS ).

[eblondel]

@elakkkk I've switch to the new URL for the 'ISTAT' id. The old platform is now identified as 'ISTAT_LEGACY'. Let me know if you can query the new endpoint

[elakkkk]

@eblondel I can see the change here on github, but I checked on R with getSDMXServiceProviders() and I can't find "ISTAT_LEGACY" ("ISTAT" has still the old URLs). I tried re-installing the package but it doesn't work. Maybe I have to do something to access it but I don't know what. Please, let me know if I have to do something in particular, and thank you for helping me!

[eblondel]

It seems a problem of R session cache. In case you have multiple R / Rstudio sessions open, maintain only one open, multiple sessions can block installation of R packages. Make sure to restart R before reinstalling rsdmx. I tried now it works

[elakkkk]

I tried restarting the session but it didn't work, I noticed that if I install the package through install.packages the provider is not updated, but installing the package through devtools::install_github("opensdmx/rsdmx") it works! I can query only the new endpoint, while the old one doesn't work. More specifically, it tries to fetch to the wrong URL. Here, an example to better explain the issue:

• with the old provider called ISTAT (which is now ISTAT_LEGACY), when downloading a dataset with id, for example, "12_60", the download path was:

 ### THIS IS THE FUNCTION WE USED BEFORE YOU ADDED THE NEW PROVIDER ###
data <- readSDMX(providerId = "ISTAT", resource = "data", flowRef="12_60", dsd =TRUE)
[rsdmx][INFO] Fetching 'http://sdmx.istat.it/SDMXWS/rest/data/12_60/all/all/' 
[rsdmx][INFO] Attempt to fetch DSD ref from dataflow description 
[rsdmx][INFO] Fetching 'http://sdmx.istat.it/SDMXWS/rest/dataflow/all/12_60/latest/' 
[rsdmx][INFO] Fetching 'http://sdmx.istat.it/SDMXWS/rest/datastructure/all/DCCV_CONSACQUA/latest/?references=children' 
[rsdmx][INFO] DSD fetched and associated to dataset!

• now, when downloading from ISTAT_LEGACY, the download path is:

data2 <- readSDMX(providerId = "ISTAT_LEGACY", resource = "data", flowRef = "12_60",  dsd = TRUE)
[rsdmx][INFO] Fetching 'http://sdmx.istat.it/SDMXWS/rest/data/12_60/all/all/' 
[rsdmx][INFO] DSD ref identified in dataset = 'IT1_DCCV_CONSACQUA_1_4' 
[rsdmx][INFO] Attempt to fetch & bind DSD to dataset 
[rsdmx][INFO] Fetching 'http://sdmx.istat.it/SDMXWS/rest/datastructure/all/IT1_DCCV_CONSACQUA_1_4/latest/?references=children' 
Not Found (HTTP 404).Error in readSDMX(providerId = providerId, providerKey = providerKey,  : 
  HTTP request failed with status: 404 

The problem is that it fetches the wrong URL: it should be http://sdmx.istat.it/SDMXWS/rest/datastructure/all/DCCV_CONSACQUA/latest/?references=children and not http://sdmx.istat.it/SDMXWS/rest/datastructure/all/IT1_DCCV_CONSACQUA_1_4/latest/?references=children .

[eblondel]

Sorry, I didn't have time to reply. Presently the new ISTAT provider is failing, sounds the service is down.

Checking the code, I realized that there was some hardcoded rule for ISTAT to force checking existence of data flow and derive datastructure from there. I now commited a fix for this.

Really sorry for the delay.

[elakkkk]

Don't worry, thank you for the update. Now everything works!
A colleague and me have just published a package to download data from Istat called "istat". Right now it works with the previous version of rsdmx (the one with just one Istat provider). We wanted to know if, and when, rsdmx package will be updated, so that the new version can be downloaded from CRAN (we managed to use it downloading it from github). Thank you again for your availability.

[eblondel]

Just submitted a rsdmx revision to CRAN. I'll let you once it is accepted.
Best

[elakkkk]

Thank you again!

[eblondel]

I got rejection (pretty normal after 1 year), essentially due to changes in R documentation requirements, broken links in doc, etc. I will review it asap and resubmit. Hopefully next week you will get it on CRAN

[eblondel]

@elakkkk I've checked now and ISTAT rsdmx tests are failing: https://esploradati.istat.it/SDMXWS gets a 404 Not found. Is it temporary?