TD 1.1 JSON-LD/RDF roundtripping example #2031

DavideRossi · 2024-07-11T18:47:40Z

Can anyone provide a working example of TD roundtripping?
Say we start with this simple example TD
Link to thingweb TD playground
Encoding to RDF looks simple, it can be done online using the JSON-LD Playground.
So far so good, we have N-Quads RDF and we can use any RDF tool.
Now I start having issues. How should I convert back from RDF to TD?
Via compacting? Via framing (and where is the TD frame definition?)
Can anyone provide a working toolchain? Online converters, code, whatever. I would just like to see this thing working.

egekorkan · 2024-07-11T19:26:04Z

@wiresio has some experience within https://github.com/eclipse-thingweb/domus-tdd-api . Could you help here?

wiresio · 2024-08-01T11:00:00Z

Hi @DavideRossi, as @egekorkan wrote you can find everything in the repo above, especially in: https://github.com/eclipse-thingweb/domus-tdd-api/blob/main/tdd/td.py and https://github.com/eclipse-thingweb/domus-tdd-api/blob/main/tdd/sparql.py

JSON-LD to RDF
Store RDF for each TD as a named graph
Retrieve named graph via SPARQL construct query
Convert N-Quads to JSON-LD
Frame JSON-LD

When you extract the corresponding pieces from our code, you should be able to roundtrip your example via JSON-LD playground (JSON-LD to RDF), local Jena installation (store RDF and carry out the SPARQL query with JSON-LD as output format), and JSON-LD playground again (framing).

wiresio · 2024-08-01T11:15:39Z

In our AID plugin, available at https://github.com/wiresio/domus-tdd-api-plugin-aid, we go one step further and carry out a format conversion:

TD JSON-LD to RDF
Store TD RDF
Retrieve via a different SPARQL construct query (TD -> AID)
Convert N-Quads
Frame with AAS context and retrieve AID JSON-LD

DavideRossi · 2024-08-01T15:02:32Z

Thanks a lot @wiresio I'm definitively going to look into that!

DavideRossi · 2024-08-05T10:51:15Z

I was able to single out the problem I'm having with roundtripping (I have some half-baked Kotlin code for a Thing Directory).
Titanium JSON-LD 1.1 Processor complains about basic_sc not being an URI even with URI validation disabled.
This does not apply to jsonld.js used by @wiresio, which is just happy with it.
But it looks like Titanium is right, basic_sc is not an URI at all (IRIs, at a minimum, must have a ':' somewhere) and subjects in RDF (if memory serves me well) must be IRIs or blank nodes.
So the question is: is this a bug in Titanium, is the TD I started with wrong (not well formed), is it an error in the TD specs (stating explicitly that TD can be non-well formed), or is there something else I'm missing?

wiresio · 2024-08-05T11:22:11Z

Please see here: #1193

DavideRossi · 2024-08-05T20:29:34Z

Thanks @wiresio , but I don't know if that is the same problem. If I got it right (and this may very well not be the case) I should replace the TD's @context with the latest td-context-1.1.jsonld in the repo.
But still basic_sc is not an IRI.
A solution could be adding a @vocab to the context (arguably using the TD's URL in the directory) but to me it looks like I'm just fiddling with it.
Which is the correct way of dealing with that?

Post scriptum: I realized the @vocab solution cannot work, it will transform basic_sc to an IRI when used as a subject but, obviously, it will be left untouched when used as an object so that's clearly a no-go.

DavideRossi · 2024-08-05T22:11:18Z

OK, my previous message was clearly me misunderstanding what's going on here.
Correct me if I'm wrong: the new td-context-1.1.jsonld introduces a new property to fix the security-related round trip problems.
Which means that replacing the original TD's @context with this new one I should get a correct conversion to RDF.
Well I did it, and still Titanium complains about basic_sc being not well formed.
What am I missing this time? Am I supposed to further change something in the TD?

Notice, it's not just Titanium. I tried the RDF conversion with the JSON-LD Playground as well and the result is:

<urn:uuid:0804d572-cce8-422a-bb7c-4412fcd56f06> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <https://www.w3.org/2019/wot/td#Thing> .
<urn:uuid:0804d572-cce8-422a-bb7c-4412fcd56f06> <https://www.w3.org/2019/wot/td#definesSecurityScheme> _:b0 .
<urn:uuid:0804d572-cce8-422a-bb7c-4412fcd56f06> <https://www.w3.org/2019/wot/td#description> "Thing Description for a Lamp thing"@en .
<urn:uuid:0804d572-cce8-422a-bb7c-4412fcd56f06> <https://www.w3.org/2019/wot/td#hasActionAffordance> _:b1 .
<urn:uuid:0804d572-cce8-422a-bb7c-4412fcd56f06> <https://www.w3.org/2019/wot/td#hasEventAffordance> _:b3 .
<urn:uuid:0804d572-cce8-422a-bb7c-4412fcd56f06> <https://www.w3.org/2019/wot/td#hasPropertyAffordance> _:b6 .
<urn:uuid:0804d572-cce8-422a-bb7c-4412fcd56f06> <https://www.w3.org/2019/wot/td#hasSecurityConfiguration> "basic_sc" .
<urn:uuid:0804d572-cce8-422a-bb7c-4412fcd56f06> <https://www.w3.org/2019/wot/td#title> "MyLampThing"@en .
_:b0 <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <https://www.w3.org/2019/wot/security#BasicSecurityScheme> .
_:b0 <https://www.w3.org/2019/wot/security#in> "header" .
_:b1 <https://www.w3.org/2019/wot/td#hasForm> _:b2 .
_:b1 <https://www.w3.org/2019/wot/td#isIdempotent> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
_:b1 <https://www.w3.org/2019/wot/td#isSafe> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
_:b1 <https://www.w3.org/2019/wot/td#name> "toggle" .
_:b2 <https://www.w3.org/2019/wot/hypermedia#forContentType> "text/plain" .
_:b2 <https://www.w3.org/2019/wot/hypermedia#hasOperationType> <https://www.w3.org/2019/wot/td#invokeAction> .
_:b2 <https://www.w3.org/2019/wot/hypermedia#hasTarget> "https://mylamp.example.com/toggle"^^<http://www.w3.org/2001/XMLSchema#anyURI> .
_:b3 <https://www.w3.org/2019/wot/td#hasForm> _:b4 .
_:b3 <https://www.w3.org/2019/wot/td#hasNotificationSchema> _:b5 .
_:b3 <https://www.w3.org/2019/wot/td#name> "overheating" .
_:b4 <https://www.w3.org/2019/wot/hypermedia#forContentType> "text/plain" .
_:b4 <https://www.w3.org/2019/wot/hypermedia#forSubProtocol> "longpoll" .
_:b4 <https://www.w3.org/2019/wot/hypermedia#hasOperationType> <https://www.w3.org/2019/wot/td#subscribeEvent> .
_:b4 <https://www.w3.org/2019/wot/hypermedia#hasTarget> "https://mylamp.example.com/oh"^^<http://www.w3.org/2001/XMLSchema#anyURI> .
_:b5 <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <https://www.w3.org/2019/wot/json-schema#StringSchema> .
_:b5 <https://www.w3.org/2019/wot/json-schema#readOnly> "true"^^<http://www.w3.org/2001/XMLSchema#boolean> .
_:b5 <https://www.w3.org/2019/wot/json-schema#writeOnly> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
_:b6 <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <https://www.w3.org/2019/wot/json-schema#StringSchema> .
_:b6 <https://www.w3.org/2019/wot/json-schema#readOnly> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
_:b6 <https://www.w3.org/2019/wot/json-schema#writeOnly> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
_:b6 <https://www.w3.org/2019/wot/td#hasForm> _:b7 .
_:b6 <https://www.w3.org/2019/wot/td#isObservable> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
_:b6 <https://www.w3.org/2019/wot/td#name> "status" .
_:b7 <https://www.w3.org/2019/wot/hypermedia#forContentType> "text/plain" .
_:b7 <https://www.w3.org/2019/wot/hypermedia#hasOperationType> <https://www.w3.org/2019/wot/td#readProperty> .
_:b7 <https://www.w3.org/2019/wot/hypermedia#hasTarget> "https://mylamp.example.com/status"^^<http://www.w3.org/2001/XMLSchema#anyURI> .

where you have a id-less BasicSecurityScheme (_:b0) while the thing references "basic_sc" as its hasSecurityConfiguration...

wiresio · 2024-08-09T16:03:18Z

What happens if you try:

JSON-LD -> RDF with original context
JSON-LD framing with modified context

DavideRossi · 2024-08-15T08:15:01Z

First of, @wiresio, thanks a lot for you continued assistance.
By trying to follow your advice I mixed all the possible combinations of 3 contexts: the one published at https://www.w3.org/2022/wot/td/v1.1, the latest version from the repo, and the one you're using in domus-ttd.
I tried performing all the various conversion steps using both online tools (the JSON-LD Playground and the RDF Distiller) and Java code with Titanium,
The issue with basic_sc not being an IRI is easily solved by adding a @base entry in the context (which I see you're doing in domus-ttd).
Still other issues are surfacing. After a lot of scrambling I'm inching closer but I still have a problem with strings framing, that does not depend on the version of the context used.
Here is an easily reproducible version of the problem. Let's start with a very minimal TD:

{
    "@context": "https://www.w3.org/2022/wot/td/v1.1",
    "@type": "Thing",
    "title": "MyThingTitle"
}

Notice: this TD is not valid since it's missing security related info, but I'm not interested in validation now.
Using the JSON-LD Playground and the RDF Distiller, the roundtripping works just fine.
Using Titanium I get the following end result:

{
    "@type": "Thing",
    "td:title": {
        "@value": "MyThingTitle",
        "@type": "http://www.w3.org/1999/02/22-rdf-syntax-ns#langString"
    },
    "@context": "https://www.w3.org/2022/wot/td/v1.1"
}

By trying to understand what's going on I seen that in the context you can find:

      "title": {
        "@id": "td:title",
        "@language": "en"
      },

Let's see what happens.
Both Titanium and JSON-LD playground produce the following RDF triples for the TD:

_:b0 <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <https://www.w3.org/2019/wot/td#Thing> .
_:b0 <https://www.w3.org/2019/wot/td#title> "MyThingTitle"@en .

So far so good.
Now let's look at the unframed RDF -> JSON.
Here RDF Distiller's output (that also perform compaction):

{
  "@id": "_:b0",
  "@type": "https://www.w3.org/2019/wot/td#Thing",
  "https://www.w3.org/2019/wot/td#title": {
    "@language": "en",
    "@value": "MyThingTitle"
  }
}

And that is Titanium (uncompacted):

[
    {
        "@id": "_:b0",
        "@type": [
            "https://www.w3.org/2019/wot/td#Thing"
        ],
        "https://www.w3.org/2019/wot/td#title": [
            {
                "@value": "MyThingTitle",
                "@type": "http://www.w3.org/1999/02/22-rdf-syntax-ns#langString"
            }
        ]
    }
]

Both use a value type for the title, but Titanium loses the language info and adds a @type.
It seems like, given the fact that in RDF the object is "MyThingTitle"@en (an RDF plain literal with language tags) and not just a "plain literal", Titanium decided that this is not a "basic" RDF string that can be mapped to a "basic" JSON string but it is a richer data type for which it wants to preserve information that would be lost otherwise (but it fails, given the fact that the only additional information, the language tags, is lost anyway).
The Ruby library, on the other side, decided to map the RDF string to a JSON string and maintain the language tag part by adding "@language": "en".
Now, the behavior of the Ruby library seems much more reasonable to me, yet I've not been able to find a point in the JSON-LD recommendation that explicitly states how language tags should be converted in these cases.
If I'm sure that what Titanium does is wrong, I could open an issue. I think I will try that anyway...

DavideRossi · 2024-08-15T08:48:09Z

Oh, and I forgot a bit (this thing is a bloody mess). You can force Titanium to behave like the Ruby/Javascript libs with strings by setting setUseNativeTypes to false.
But then, all your boolean properties become like this:

            "observable": {
                "@value": "false",
                "type": "xsd:boolean"
            }

So you either have messed up booleans or messed up strings!

DavideRossi · 2024-08-18T19:29:29Z

Oh well. These were two bugs in Titanium JSON-LD.
One was an issue with RDF strings that are stored as such when they have a language tag and are stored as JSON strings when they don't, and another was a compaction bug, where the type context is not updated in sub-contexts. Took me a couple of days to figure that out.
It also means that all Java-based thing directories out there should suffer from the same problem.
I wonder how @AndreaCimminoArriaga managed that in WotHive...

wiresio · 2024-09-09T09:59:47Z

@DavideRossi - sorry for the late reaction: Is the summary that you have / had challenges with the Titanium stack or do you thing that there is (still) an issue with the TD context?

DavideRossi · 2024-09-09T10:36:14Z

Yes, the issues I was experiencing were due to a couple of bugs in Titanium JSON-LD.
I have fixed them in the code I use and it works just fine.
That being said, I only tested it with rather elementary TDs, I can't rule out that other context related issues exist.

AndreaCimminoArriaga · 2024-09-19T09:01:02Z

Hi @DavideRossi, I'm a bit late for the conversation. But in the case it may help, I follow the same approach for framing as mentioned above. For the titanium issues, we tried to locate them and fix them with code after performing a regular round trip from RDF to JSON-LD 1.1. In all likelihood, there will be some cases we did not located yet but for fixing these issues we did not found another way.

So for instance, translating from RDF to JSON-LD 1.1 requires the TD to have @type : Thing. However, many times TDs do not specifically provide the type and it is assumed they are td:Things. In these cases we store meta information about the TD where we specify a certain TD has no type, and we inject the type manually when stored in the triple store. When this TD is retrieved, we translate it into JSON-LD 1.1 and then reading the meta information we modify it, for instance, removing the type if the original TD hadn't it written.

DavideRossi · 2024-09-19T09:47:00Z

Hi @DavideRossi, I'm a bit late for the conversation. But in the case it may help, I follow the same approach for framing as mentioned above. For the titanium issues, we tried to locate them and fix them with code after performing a regular round trip from RDF to JSON-LD 1.1. In all likelihood, there will be some cases we did not located yet but for fixing these issues we did not found another way.

The are issues that are hard to fix with this approach, mostly related to type management. Localized strings come to mind, but in general JSON type conversions is broken in Titanium whenever you deal with a nested framing document (as in this case).

So for instance, translating from RDF to JSON-LD 1.1 requires the TD to have @type : Thing. However, many times TDs do not specifically provide the type and it is assumed they are td:Things. In these cases we store meta information about the TD where we specify a certain TD has no type, and we inject the type manually when stored in the triple store. When this TD is retrieved, we translate it into JSON-LD 1.1 and then reading the meta information we modify it, for instance, removing the type if the original TD hadn't it written.

Yes, if you want "fully transparent round-trip" you have to resort to this kind of approaches. I'm not sure, however, about the need the keep track of things in the registration phase just to undo them for look-ups.
On one hand, clients that break because a retrieved TD has a @type it didn't have when registered were probably flawed to begin with. On the other hand, not having 1:1 mappings could be problematic for things like partial updates. I still need to wrap my head around that.

AndreaCimminoArriaga · 2024-10-04T13:37:37Z

It is true that Titanium has some issues, but most of those I struggled with come from the fact that JSON-LD 1.1 document needs to be exactly the same without differences as the original one. This is challenging because RDF is not meant for instance to have order (unless you rely on rdf:List), which makes challenging returning an array in the same order as the original TD document. Maybe, reviewing the @context of TDs may help reducing this kind of issues.

I also remember to have some issues with the security, that was translated into RDF with a URI that when going back to JSON-LD 1.1 was not translated as the original term. In these cases there is no other way to solve the issue than developing these ad-hoc translations (which I'm not found as a solution).

Finally, we need to consider that people use TDs adding some other vocabularies to improve the granularity of the data described. This also may lead to some minor issues but, maybe, this is a topic for a different issue.

DavideRossi · 2024-10-04T18:23:16Z

The security thing is again a problem with Titanium. In my fixed version it works no problem.
As for the exact same document, an easy solution is to store the JSON as an RDF literal in the triplestore and return that when it's needed.
Given that all modifications operate on the JSON representation, you just need to make sure that whenever that changes the RDF triples are modified accordingly.
At the end of the day RDF only "surfaces" with SPARQL queries, if I'm not mistaken.
But, yes, even if it could work, it surely is an inelegant solution...

AndreaCimminoArriaga · 2024-10-05T09:19:02Z

I like your idea of storing the original Json as a literal, probably it would solve more than one headache I had

DavideRossi · 2024-10-06T10:50:29Z

One of my students implemented a WOT Directory (in Kotlin) by keeping an in-memory collection of the TDs as JSON, then each time a JSON TD is changed (or a new one is created) the associated RDF graph is reconstructed. You can check it at https://github.com/dvgniele/WoTerFlow.
In its current implementation, however, the triplestore is the only persistent structure. Which means that when re-starting the directory, it re-creates the in memory collection with a set of RDF -> JSON conversion, which is broken because he used the bugged version of Titanium. This is where we discussed to store the JSON in the RDF graph...
I do not like Kotlin's available IDEs so I do not plan to fix it (unless a willing new student appears).

github-actions bot added the needs-triage Automatically added to new issues. TF should triage them with proper labels label Jul 11, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TD 1.1 JSON-LD/RDF roundtripping example #2031

TD 1.1 JSON-LD/RDF roundtripping example #2031

DavideRossi commented Jul 11, 2024

egekorkan commented Jul 11, 2024

wiresio commented Aug 1, 2024

wiresio commented Aug 1, 2024

DavideRossi commented Aug 1, 2024 •

edited

Loading

DavideRossi commented Aug 5, 2024

wiresio commented Aug 5, 2024

DavideRossi commented Aug 5, 2024 •

edited

Loading

DavideRossi commented Aug 5, 2024

wiresio commented Aug 9, 2024

DavideRossi commented Aug 15, 2024

DavideRossi commented Aug 15, 2024 •

edited

Loading

DavideRossi commented Aug 18, 2024

wiresio commented Sep 9, 2024

DavideRossi commented Sep 9, 2024

AndreaCimminoArriaga commented Sep 19, 2024

DavideRossi commented Sep 19, 2024 •

edited

Loading

AndreaCimminoArriaga commented Oct 4, 2024

DavideRossi commented Oct 4, 2024

AndreaCimminoArriaga commented Oct 5, 2024

DavideRossi commented Oct 6, 2024

TD 1.1 JSON-LD/RDF roundtripping example #2031

TD 1.1 JSON-LD/RDF roundtripping example #2031

Comments

DavideRossi commented Jul 11, 2024

egekorkan commented Jul 11, 2024

wiresio commented Aug 1, 2024

wiresio commented Aug 1, 2024

DavideRossi commented Aug 1, 2024 • edited Loading

DavideRossi commented Aug 5, 2024

wiresio commented Aug 5, 2024

DavideRossi commented Aug 5, 2024 • edited Loading

DavideRossi commented Aug 5, 2024

wiresio commented Aug 9, 2024

DavideRossi commented Aug 15, 2024

DavideRossi commented Aug 15, 2024 • edited Loading

DavideRossi commented Aug 18, 2024

wiresio commented Sep 9, 2024

DavideRossi commented Sep 9, 2024

AndreaCimminoArriaga commented Sep 19, 2024

DavideRossi commented Sep 19, 2024 • edited Loading

AndreaCimminoArriaga commented Oct 4, 2024

DavideRossi commented Oct 4, 2024

AndreaCimminoArriaga commented Oct 5, 2024

DavideRossi commented Oct 6, 2024

DavideRossi commented Aug 1, 2024 •

edited

Loading

DavideRossi commented Aug 5, 2024 •

edited

Loading

DavideRossi commented Aug 15, 2024 •

edited

Loading

DavideRossi commented Sep 19, 2024 •

edited

Loading