Thoughts about JSON

[mosheduminer]

I've been thinking about the prerequisites to JSON (de)serialization to/from data structures in virgil. What I would like is a library like serde and serde_json in rust, where data can be deserialized directly into a struct, and serialized directly from a struct.

However, it seems to me that I'm virgil the only way to deserialize from json would be to deserialize into a recursive ADT of hashmaps / arrays / allowed primitives. If I wanted to get the data into (or out of) a well defined data structure though, i would to write additional code to connect the ADT structure to a class (or any other virgil structure).

What this leads me to, is the fact that there is no reflection in virgil, even in compile time code. If it were possible to allow reflection at compile time, I imagine it would open up static metaprogramming such as is necessary for a library like serde. However, I know that reflection is listed as something that virgil lacks, and I would like to know if the reasons for that (I assume performance? Are there also other considerations) also preclude compile time reflection, and if not, whether you would consider adding it to your roadmap. I understand that virgil is in a sense an expiriment (?), and I of course do not wish to imply that there is any onus on you to add a particular feature.

[k-sareen]

I think you'll find that the proposed WIP struct data structures precisely do what you want -- having a well defined data structure over an array of bytes essentially. It's supposed to be used for low-level code such as networking and interacting with the kernel, and it could be used for JSON parsing as well.

@titzer can correct me if I'm wrong, but I feel like it should be possible with structs (assuming I understand them correctly)

EDIT: The above is based on the assumption that you know the structure of the JSON data, If you're trying to make a general library for it, then I think structs may not be enough? I'm not sure.

[mosheduminer]

I'm sorry, but why) I don't understand why structs would be better for this purpose? My problem is not wanting to write custom code for each data structure I want to convert to/from json, a problem which would maybe be solved with appropriate compile time features. But I don't see why structs would be better for this?

[k-sareen]

See comment below; I am/was mistaken -- it may not work unless you have a fixed size.

[k-sareen]

Actually on second thought -- structs may not be sufficient as they assume a fixed size? I'm a bit hazy on the capabilities of structs, but if they are fixed size only, then a string format like JSON, where each instance of the same JSON structure may have different lengths, may not work.

[titzer]

I've been thinking about the prerequisites to JSON (de)serialization to/from data structures in virgil. What I would like is a library like serde and serde_json in rust, where data can be deserialized directly into a struct, and serialized directly from a struct.

However, it seems to me that I'm virgil the only way to deserialize from json would be to deserialize into a recursive ADT of hashmaps / arrays / allowed primitives. If I wanted to get the data into (or out of) a well defined data structure though, i would to write additional code to connect the ADT structure to a class (or any other virgil structure).

What this leads me to, is the fact that there is no reflection in virgil, even in compile time code. If it were possible to allow reflection at compile time, I imagine it would open up static metaprogramming such as is necessary for a library like serde. However, I know that reflection is listed as something that virgil lacks, and I would like to know if the reasons for that (I assume performance? Are there also other considerations) also preclude compile time reflection, and if not, whether you would consider adding it to your roadmap. I understand that virgil is in a sense an expiriment (?), and I of course do not wish to imply that there is any onus on you to add a particular feature.

The motivation for not (yet) having reflection has been primarily about memory usage and optimizability of Virgil code. Reflection requires having source-level metadata connected to program values, and that metadata takes a lot of space. It can also prevent a lot of the static analysis the compiler does to optimize and remove dead code.

That said, reflection only at compile-time would restrict those problems to a phase that can be completed, after which optimization can happen. That's something I could see as a future feature.

Failing that, there are a number of non-reflective things that you can try first.

1. Use dynamically typed (or "stringly"-typed) objects to represent JSON, so that you can write code like:

type JSONVal {
 ...
 def toObj() -> JSONVal.Obj;
 def toInt() -> int;
}
var x = deserializeJSON(input);
var y = x["field1"].toObj();
var z = y["field2"].toInt();

2. Minimize the amount of manual code you'd have to write by designing a serializer combinator library:

class MyClass(var x: int, var y: int) {
}
// deserializes directly into constructor of MyClass
def deserializeMyClass = Deserializer.new2<MyClass>(MyClass.new, Deserializer.i32("x"), Deserializer.i32("y"));
// serializes using MyClass.field accessor functions
def serializeMyClass = Serializer.new2<MyClass>(Serializer.i32(MyClass.x), Serializer.i32(MyClass.y));

The above is a bit verbose, but you could:

3. Use the Virgil parser (from src/aeneas) as a library in another tool to process Virgil source code and generate more Virgil source that serializes/deserializes a type.

[k-sareen]

Oh... Maybe that syntax is only if you don't destructure the case? It's been a while, but I'm fairly certain that x: Case is valid syntax.

Edit: Yeah see https://github.com/titzer/virgil/blob/master/aeneas/src/mach/MachBackend.v3#L672-L675. It may be that you can't destructure the case.

[mosheduminer]

I'll try tomorrow morning then... It's late where I am.

[mosheduminer]

Worst case I can cast, as you suggested.

[k-sareen]

For what it's worth, the following does work:

type A {
        case B(v: int);
        case C(c: long);
}

def main() {
        def s = A.B(10);
        match (s) {
                x: A.B => {
                        System.puti(x.v);
                        System.puts("\n");
                }
                _ => {
                        System.error("Error!", "Unexpected case");
                }
        }
}

[mosheduminer]

I see, so interestingly, only this x: JsonValue.Number works for me:

match (json) {
	x: JsonValue.Number => return x.i32();
	_ => return Result.Err("Expected to find Number");
}

but x: Number and x: JsonValue.Number(v) do not. I would expect that at least x: Number should work though?

[titzer]

The syntax for match is either a new variable binding with a type or a destructure:

type T {
  case X(a: int) { }
  case Y(b: int) { }
}

def m(v: T) {
  match (v) { // query, cast, bind new variable
    x: T.X => { }
    x: T.Y => { }
  }
  match (v) { // destructure
    X(a) => { }
    Y(b) => { }
  }
}

There are routines to parse floats inside the compiler (look for NumberParser.v3 and FloatSketch.v3). They have just not been pulled out to a library yet.

[mosheduminer]

Thanks! It works now 🥳.

~/virgil/parser-combinators$ make test
rm -f Test
v3c-x86-64-linux Json.v3 PC.v3 Test.v3 ../virgil/lib/util/*.v3 ../virgil/lib/test/*.v3 ./extracted/*.v3
~/virgil/parser-combinators$ ./Test
##>3 unit tests
##+parsing
##-ok
##+deserializing
##-ok
##+fail_deserializing
##-ok

[mosheduminer]

I remember at some point in the past (can't find it now) you suggested that the parser combinator (and the JSON parser too?) be contributed to this repo as a PR. Are you still interested, and if so do you have any guidelines / suggestions for that?

[titzer]

Nice!

Let's do the JSON parsing utility as a first PR, then we can design parser combinators as a follow-up. Wdyt?

[mosheduminer]

Well, my JSON parsing is implemented on top of my parser combinator system, so I don't think that ordering will work 😃.

[mosheduminer]

If you prefer an independent JSON parser I can maybe work on that though.