Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding BCO-DMO specific metadata to datapackage.json #1

Closed
ashepherd opened this issue Apr 9, 2019 · 5 comments
Closed

Adding BCO-DMO specific metadata to datapackage.json #1

ashepherd opened this issue Apr 9, 2019 · 5 comments
Assignees

Comments

@ashepherd
Copy link
Contributor

ashepherd commented Apr 9, 2019

See also:

PROPOSAL

  1. If adding JSON-LD support is of interest (frictionlessdata/specs#218), define a namespace within frictionlessdata.io domain for potentially serving machine-readable descriptions of Frictionlessdata classes and properties. (NOTE: not required)

  2. Include an @context at the start of any datapackage.json:

 {
  "@context": {
    "@vocab": "http://schema.frictionlessdata.io/v1.0/data-package/",
    "odo": "http://ocean-data.org/schema/",
    "units": "http://vocab.nerc.ac.uk/collection/P06/current/",
    "param": "http://vocab.nerc.ac.uk/collection/P01/current/",
    "resources": {
      "@id": "odo:dataFile",
      "@context": {
        "fields": {
          "@id": "odo:storesValuesFor"
        }
      }
    }
  },
  ... JSON contents go here ...
}

We use the @vocab to specify that all the terms in the JSON document are defined in the JSON-LD context provided at http://schema.frictionlessdata.io/v1.0/data-package/

odo, units and param defines other RDF vocabularies that will be used to define non-default terms in the document.

resources is redefined from the default vocabulary to a term in the odo vocabulary where its scoped property fields is also defined to be in the odo vocabulary. This technique saves time from having to duplicating the same value twice in two different fields. Effectively, it means that resources should be interpreted as both 1) resources from the @vocab schema, and 2) as odo:storesValuesFor from the odo schema.

For a full example we modify the frictionlessdata:example-data-packages/periodic-table below:

{
  "@context": {
    "@vocab": "http://schema.frictionlessdata.io/v1.0/data-package/",
    "odo": "http://ocean-data.org/schema/",
    "units": "http://vocab.nerc.ac.uk/collection/P06/current/",
    "param": "http://vocab.nerc.ac.uk/collection/P01/current/",
    "resources": {
      "@id": "odo:dataFile",
      "@context": {
        "fields": {
          "@id": "odo:storesValuesFor"
        }
      }
    }
  },
  "name": "period-table",
  "title": "Periodic Table",
  "profile": "tabular-data-package",
  "licenses": [{
    "id": "CC0-1.0",
    "title": "CC0 1.0",
    "url": "https://creativecommons.org/publicdomain/zero/1.0/"
  }],
  "resources": [{
    "@type": "odo:DataFile",
    "path": "data.csv",
    "name": "data",
    "profile": "tabular-data-resource",
    "format": "csv",
    "mediatype": "text/csv",
    "encoding": "UTF-8",
    "schema": {
      "fields": [{
          "name": "atomic number",
          "type": "integer",
          "format": "default"
        },
        {
          "name": "symbol",
          "type": "string",
          "format": "default",
          "odo:unitOfMeasure": {
              "@id": "units:UUUU"
            }
        },
        {
          "name": "name",
          "type": "string",
          "format": "default"
        },
        {
         "@type": "param:TTPBKG01",
          "name": "atomic mass",
          "type": "number",
          "format": "default",
          "odo:unitOfMeasure": {
              "@type": "odo:UnitOfMeasure",
              "odo:title": "Daltons",
              "odo:unitSymbol": "Da"
            }
        },
        {
          "name": "metal or nonmetal?",
          "type": "string",
          "format": "default"
        }
      ]
    }
  }]
}

To see the RDF triples resulting from this JSON-LD, go to the JSON-LD Playground:

  1. Set the Options > Processing Mode to JSON-LD 1.1

Screen Shot 2019-04-09 at 4 27 56 PM

2. Paste the above JSON-LD into the textarea found at the `JSON-LD Input` tab 3. Click the `Table` tab below the `JSON-LD Input`

Screen Shot 2019-04-09 at 4 27 57 PM

Using this method, we can expand the datapackage.json, using RDF vocabulary terms to describe our Dataset's funding awards, projects, authors and contributors, deployments, parameters, instruments, supplemental documents, related publications, etc.

{
  "@context": {
    "@vocab": "http://schema.frictionlessdata.io/v1.0/data-package/",
    "odo": "http://ocean-data.org/schema/",
    "units": "http://vocab.nerc.ac.uk/collection/P06/current/",
    "param": "http://vocab.nerc.ac.uk/collection/P01/current/",
    "resources": {
      "@id": "odo:dataFile",
      "@context": {
        "fields": {
          "@id": "odo:storesValuesFor"
        }
      }
    }
  },
  "name": "period-table",
  "title": "Periodic Table",
  "profile": "tabular-data-package",
  "licenses": [{
    "id": "CC0-1.0",
    "title": "CC0 1.0",
    "url": "https://creativecommons.org/publicdomain/zero/1.0/"
  }],
  "resources": [{
    "@type": "odo:DataFile",
    "path": "data.csv",
    "name": "data",
    "profile": "tabular-data-resource",
    "format": "csv",
    "mediatype": "text/csv",
    "encoding": "UTF-8",
    "schema": {
      "fields": [{
          "name": "atomic number",
          "type": "integer",
          "format": "default"
        },
        {
          "name": "symbol",
          "type": "string",
          "format": "default",
          "odo:unitOfMeasure": {
              "@id": "units:UUUU"
            }
        },
        {
          "name": "name",
          "type": "string",
          "format": "default"
        },
        {
         "@type": "param:TTPBKG01",
          "name": "atomic mass",
          "type": "number",
          "format": "default",
          "odo:unitOfMeasure": {
              "@type": "odo:UnitOfMeasure",
              "odo:title": "Daltons",
              "odo:unitSymbol": "Da"
            }
        },
        {
          "name": "metal or nonmetal?",
          "type": "string",
          "format": "default"
        }
      ]
    }
  }],
  "odo:abstract": "Textual abstract goes here...",
  "odo:datasetTitle": "Official dataset title goes here...",
  "odo:authoredBy":[{
    "@type": "odo:AgentRole",
    "odo:roleType": ["odo:Role_PrincipalInvestigator, "odo:Role_Contact"],
    "odo:performedBy": {
      "@type": "odo:Person",
      "odo:familyName": "Morris",
      "odo:givenName": "Robert",
      "odo:honorific": "Dr."
    },
    "odo:forOrganization": { "@id": "https://www.bco-dmo.org/affiliation/27" }
  }],
  "odo:fromInstrument": [{
    "@type": "odo:Instrument",
    "@id": "http://lod.bco-dmo.org/id/instrument/488",
    "odo:title": "CTD Seabird 9 plus",
    "odo:description": "Seabird 9plus CTD with temperature and conductivity sensors."
  },
  {
    "@type": "odo:Instrument",
    "@id": "http://lod.bco-dmo.org/id/instrument/685",
    "odo:title": "Thermo Fisher (San Jose, Ca) linear ion trap –Orbitrap (LTQ-OT) hybrid tandem mass spectrometer",
    "odo:description": "\"The hybrid Fourier Transform (FT) mass spectrometer(MS) combines a linear ion trap MS and the Orbitrap mass analyzer. Ions generated by API are collected in the LTQ XL followed by axial ejection to the C-shaped storage trap which is used to store and collisionally cool ions before injection into the orbital trap. The ions transferred from the C-Trap are captured in the orbital trap by rapidly increasing the electric field and the detection of the image current from coherent ion packets takes place after the voltages have stabilized. Signals from each of the orbital trap outer electrodes are amplified and transformed into a frequency spectrum by fast Fourier transformation which is finally converted into a mass spectrum.\" (From 
 Fisher Scientific)"
  }],
  ...
}
@ashepherd
Copy link
Contributor Author

@adyork let's review at some point

@lwinfree
Copy link
Collaborator

Hi @ashepherd & @adyork! We are starting to check out this issue and are wondering if there are specific pain points here that you need help with? Or is this issue more documentation for the goal of eventually adding this metadata during the pilot? Thanks in advance for any clarification :-)

@ashepherd
Copy link
Contributor Author

Hi @lwinfree! Maybe just discussion about OKF plans for adding JSON-LD support? If it was of interest and OKF wanted help, I'd be happy to help!

I just want to make sure we aren't doing anything that conflicts with OKF plans by using JSON-LD, or if there weren't plans to add support, maybe our office rethinks using it here in favor of inserting a property at the top-level of the JSON that won't ever conflict with OKF future plans. Something like where we insert odo: as a top-level field, and then put all our our metadata underneath that as opposed to embedding it within the frictionless fields.

Overall, our preference would be for JSON-LD support so that both sets of information can live side-by-side without having to duplicate content (re: Frictionlessdata fields to odo:DatasetParameter)

Example:

{
   "name":"period-table",
   "title":"Periodic Table",
   "profile":"tabular-data-package",
   "licenses":[
      {
         "id":"CC0-1.0",
         "title":"CC0 1.0",
         "url":"https://creativecommons.org/publicdomain/zero/1.0/"
      }
   ],
   "resources":[
      {
         "path":"data.csv",
         "name":"data",
         "profile":"tabular-data-resource",
         "format":"csv",
         "mediatype":"text/csv",
         "encoding":"UTF-8",
         "schema":{
            "fields":[
               {
                  "name":"atomic number",
                  "type":"integer",
                  "format":"default"
               },
               {
                  "name":"symbol",
                  "type":"string",
                  "format":"default"
               },
               {
                  "name":"name",
                  "type":"string",
                  "format":"default"
               },
               {
                  "name":"atomic mass",
                  "type":"number",
                  "format":"default"
               },
               {
                  "name":"metal or nonmetal?",
                  "type":"string",
                  "format":"default"
               }
            ]
         }
      }
   ],
   "odo:":{
      "odo:abstract":"Textual abstract goes here...",
      "odo:datasetTitle":"Official dataset title goes here...",
      "odo:authoredBy":[
         {
            "@type":"odo:AgentRole",
            "odo:roleType":[
               "odo:Role_PrincipalInvestigator",
               "odo:Role_Contact"
            ],
            "odo:performedBy":{
               "@type":"odo:Person",
               "odo:familyName":"Morris",
               "odo:givenName":"Robert",
               "odo:honorific":"Dr."
            },
            "odo:forOrganization":{
               "@id":"https://www.bco-dmo.org/affiliation/27"
            }
         }
      ],
      "odo:storesValuesFor":[
         {
            "@type":"odo:DatasetParameter",
            "odo:name":"atomic number",
            "odo:dataType":"xsd:int"
         },
         {
            "@type":"odo:DatasetParameter",
            "odo:name":"symbol",
            "odo:dataType":"string",
            "odo:unitOfMeasure":{
               "@id":"units:UUUU"
            }
         },
         {
            "@type":"odo:DatasetParameter",
            "odo:name":"name",
            "odo:dataType":"xsd:string"
         },
         {
            "@type":[
               "odo:DatasetParameter",
               "param:TTPBKG01"
            ],
            "odo:name":"atomic mass",
            "odo:dataType":"xsd:float",
            "odo:unitOfMeasure":{
               "@type":"odo:UnitOfMeasure",
               "odo:title":"Daltons",
               "odo:unitSymbol":"Da"
            }
         },
         {
            "@type":"odo:DatasetParameter",
            "odo:name":"metal or nonmetal?",
            "odo:dataType":"xsd:string"
         }
      ],
      "odo:fromInstrument":[
         {
            "@type":"odo:Instrument",
            "@id":"http://lod.bco-dmo.org/id/instrument/488",
            "odo:title":"CTD Seabird 9 plus",
            "odo:description":"Seabird 9plus CTD with temperature and conductivity sensors."
         },
         {
            "@type":"odo:Instrument",
            "@id":"http://lod.bco-dmo.org/id/instrument/685",
            "odo:title":"Thermo Fisher (San Jose, Ca) linear ion trap –Orbitrap (LTQ-OT) hybrid tandem mass spectrometer",
            "odo:description":"The hybrid Fourier Transform (FT) mass spectrometer(MS) combines a linear ion trap MS and the Orbitrap mass analyzer. Ions generated by API are collected in the LTQ XL followed by axial ejection to the C-shaped storage trap which is used to store and collisionally cool ions before injection into the orbital trap. The ions transferred from the C-Trap are captured in the orbital trap by rapidly increasing the electric field and the detection of the image current from coherent ion packets takes place after the voltages have stabilized. Signals from each of the orbital trap outer electrodes are amplified and transformed into a frequency spectrum by fast Fourier transformation which is finally converted into a mass spectrum. (From Fisher Scientific)"
         }
      ]
   }
}

@cschloer
Copy link

@ashepherd Do you feel like it's okay to close this issue given our decision to add custom metadata in the datapackage.json by prefixing with bcodmo:?

@ashepherd
Copy link
Contributor Author

ashepherd commented Nov 17, 2019 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants