fq and changes in fq:s gojq fork

[wader]

Hello! not really and issue but i hope it's ok to use an issue to discuss and maybe break things out into other PR or issues if needed.

I've finally i've published my gojq based project some days ago that i've been working on for the last year. It's called fq and is result of me wanting a tool to make it easier to inspect, debug, query and work with various binary format. I've experimented with other query languages and implementation languages than go even before that but i know that i really wanted jq as it really felt like a good fit for working with all kind of tree structured data. So when i found gojq i got very excited and with lots of trying and work i think i have something close to what i imagined and it has already help me solve all kind of tasks.

So what i wanted to discuss is various changes to gojq that i've made to make fq possible. I don't have any expectations to get any of them merged but i hope at least some of them might be interesting to merge or partially merge.

The changes can be found in the "fq" branch of my gojq fork https://github.com/wader/gojq/commits/fq

Here are the changes:

• "Query marhal/unmarshal support" Is used to do AST-rewrites to implement auto completion and "slurp" in the REPL.
• "WIP: add scopedump function" will probably get removed, ignore for now
• "Add scope function to get all functions and variables in scope" Is used to implement completion
• "Add JQValue interface to support custom values" Introduces a "JQValue" interface which makes it possible for any type to behave as a jq value. This is what fq uses to make a "decode value" be usable in jq expressions and then later when needed be used to get byte ranges, actual/symbolic representation etc, it also makes it possible to be "lazy" and only do slow things or IO if needed etc. This is maybe the third or forth rewrite of this change and i think it looks kind of sane now but there are some murky bits like how updates should work and advanced slicing. I think something like JQValue in gojq would open up for many other interesting uses of gojq. If your not interested at all i think it would help me greatly to a least look into refactor out most type casts into toObject, toString, ... . It also makes PathValue public to be used by JQValueEach.
• "Add binary, octal and hex literals" Messy to upstream i guess? maybe could be done with optional parser hooks for literals somehow?
• "Add bit shift and logical operators" I thought i would use bit operations often but i haven't. Considering removing this and turn them into normal internal functions instead.
• "Rename fq gojq fork to github.com/wader/gojq" I do this because i have a private versions of fq that implements various proprietary decoders so it has fq as submodule. I didn't managed to get this setup working using replace in go.mod so this seemed simpler.

And of course i'm more than happy to talk more about fq and help understand how it can be used or how to write decoders etc. It's lacking a lot in documentation at the moment.

And lastly once again i would like to thank you for working on gojq. It has been a pleasure to work with and i really appreciate the engineering put into it. It has been very stable and also fun and educational experience.

Have a great holiday and hope for more nice jq things next year!

[itchyny]

First of all, congratulation for releasing fq after hard work for months. The tool looks pretty useful. I have been working on a binary editor called bed and also contributed a bit to kaitai struct, but I didn't come up with this idea. I searched for gojq issues you opened, looked into the commits of your fork, and see how it was hard to make gojq work to achieve your goal.
Currently gojq is fairly stable and I don't plan to add more features at this moment. You may want my thoughts to JQValue especially. Honestly speaking this looks too difficult to maintain for me and is unlikely to merge. The interface is defined for dealing your fq decode value structs, but others may want different method sets. For example, using matrix struct for applying math functions in jq way. Also, I don't want to spread out JQValue interface. The problem of Golang interface is that users cannot implement for structs imported from library. I've seen a blog using gojq against k8s structs. The author converts the struct to interface{}, which is the only and correct way, but if gojq accepts JQValue interface, they may want k8s library structs implement the interface. But this situation is not what I hope. The last problem I concern is, jq query is not only for querying but also for constructing. I like this complementarity. The JQValue interface can be used for querying but does not help people want to construct their own structs. I understand current gojq interface does not work with the laziness of binary file decoders. But this issue happens in very limited situation, and in most cases using interface{} is much more useful for both gojq users and the maintainer.
I'm honored that my package helps various tools to happen. I like the idea of fq and I realized how useful REPL is. Good luck with your work!

[wader]

First of all, congratulation for releasing fq after hard work for months. The tool looks pretty useful. I have been working on a binary editor called bed and also contributed a bit to kaitai struct, but I didn't come up with this idea.

Thanks! Yes was lot of work but mostly very interesting and i've learned a lot. I thought it would be a couple of months to make something that felt useable by others, took one year :)

Yes think i saw bed once when i looked thru your other projects so i assumed you might find fq useful. I have some vague plans to support interpreting basic kaitai definitions and maybe something else on top of the decode API for fq, will see how it goes. Also been experimenting with decoder DSL:s in jq, tricky but possible i think.

I searched for gojq issues you opened, looked into the commits of your fork, and see how it was hard to make gojq work to achieve your goal. Currently gojq is fairly stable and I don't plan to add more features at this moment. You may want my thoughts to JQValue especially. Honestly speaking this looks too difficult to maintain for me and is unlikely to merge. The interface is defined for dealing your fq decode value structs, but others may want different method sets. For example, using matrix struct for applying math functions in jq way. Also, I don't want to spread out JQValue interface.

Yes fully understand that. The JQValue addition was by far the hardest to get working and i've probably rewritten it in different way 3-4 times to make it do all things i wanted. For example i really wanted gojq to handle the values as "shallow" as possible to preserve the original types, that way you can usually use all standard functions etc and get a JQValue out the other end.

Your probably right that the current JQValue interface is quite biased towards fq:s usage. It would be interesting to try some other usages to make it more generic. Would you be interested to include some of the refactorings like toString, toBoolean etc that is part of the JQValue commit? that i think would make the task easier for me to rebase on gojq main.

The problem of Golang interface is that users cannot implement for structs imported from library. I've seen a blog using gojq against k8s structs. The author converts the struct to interface{}, which is the only and correct way, but if gojq accepts JQValue interface, they may want k8s library structs implement the interface. But this situation is not what I hope. The last problem I concern is, jq query is not only for querying but also for constructing. I like this complementarity. The JQValue interface can be used for querying but does not help people want to construct their own structs. I understand current gojq interface does not work with the laziness of binary file decoders. But this issue happens in very limited situation, and in most cases using interface{} is much more useful for both gojq users and the maintainer.

I haven't thought about it from library authors prespective, thanks for the link, i will surely read it to try understand the issue. Construction i haven't thought about either, you mean somehow allow JQValue to be used in literals or define new literals? (interesting idea :)

Yes i agree, implementing a JQValue or something similar is no easy task i've noticed, you have to be quite careful to re-implement a lot of details to make things feel normal to a user.

How do you feel about the query AST additions? I first tried to do a REPL with completion mostly in go, but it got very messy so i ended up calling out to helper functions in jq and at some point it just felt better to switch it around. And so far im very happy with the result, the slurp support was interesting to do, but i doubt many will use it, but felt hacky without it (1,2,3 | repl would start 3 REPLs).

I'm honored that my package helps various tools to happen. I like the idea of fq and I realized how useful REPL is. Good luck with your work!

Thanks, and i'm very grateful! without it and your helpfulness I don't think fq would have existed today. And please let me know if you have any other questions about fq.

BTW, if you didn't notice fq do support JSON as input also if you want to use the REPL to play around with completion and iterate on some query.

[itchyny]

How do you feel about the query AST additions? I first tried to do a REPL with completion mostly in go, but it got very messy so i ended up calling out to helper functions in jq and at some point it just felt better to switch it around.

If you mean JSON tags, I can't include this neither. I don't ensure the compatibility of the query syntax tree and JSON marshalization can break between versions. Actually the query of v0.12.5 and v0.12.6 are slightly different. Using JSON for completion of query is a quite unique idea and very rare use case. The JSON tags allow users to store or transmit the query by JSON, but query string is more robust against package updates.

[wader]

If you mean JSON tags, I can't include this neither. I don't ensure the compatibility of the query syntax tree and JSON marshalization can break between versions. Actually the query of v0.12.5 and v0.12.6 are slightly different. Using JSON for completion of query is a quite unique idea and very rare use case. The JSON tags allow users to store or transmit the query by JSON, but query string is more robust against package updates.

Yes agree using JSON tags on structs in gojq for this is not a good idea, as mentioned in #123, this was more or quick way to to get access to the AST to see if it was a possible way of doing a nice REPL in jq.
I've thought a bit how something like this could be done in a more stable way but no good ideas yet. Define a more stable query-JSON structure somehow? or make the structure "opaque" and provide a jq API (something like https://github.com/wader/fq/blob/master/pkg/interp/query.jq) for doing specific AST transformations? for now i think i will keep it in sync with gojq. But please don't hesitate changing things just because i relay on it.

Happy new year btw!

[wader]

BTW have i given you credit correctly for the gojq parts i forked for fq? (some code in https://github.com/wader/fq/tree/master/internal/gojqextra and https://github.com/wader/fq/blob/master/internal/colorjson/encoder.go is based on cli/encoder.go)