Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

json-to-dhall #878

Closed
antislava opened this issue Apr 2, 2019 · 15 comments
Closed

json-to-dhall #878

antislava opened this issue Apr 2, 2019 · 15 comments

Comments

@antislava
Copy link
Collaborator

antislava commented Apr 2, 2019

Implemented a simple json-to-dhall tool: master...antislava:antislava/json-to-dhall.
Haddock documentation following the structure of dhall-to-json is a bit rough but reasonably complete in Main.hs (cabal new-haddock exe:json-to-dhall works in dhall-json's nix-shell).

The conversion function is relatively straightforward but I had to put some thought into what the default behavior should be (in particular with regards to unions and records).

So far tested it only on simple cases (listed in the documentation). For a more "real-life" example, check out bower.dhall.

Related discussions:

@Profpatsch
Copy link
Member

Does dhall-to-json(json-to-dhall(x)) == x hold?

@antislava
Copy link
Collaborator Author

antislava commented Apr 2, 2019

@Profpatsch json-to-dhall(t, dhall-to-json(x : t)) == x, where t is the Dhall type (schema) which is the first argument to json-to-dhall, the second being the actual JSON data, should hold in most cases. Proper proof/tests will be the next step, but it does already hold for the provided bower.dhall/bower.json example (up to the key re-ordering, obviously) and in basic cases in the Main.hs documentation section:

json-to-dhall Bool <<< 'true' | dhall-to-json
json-to-dhall Bool <<< 'false' | dhall-to-json
json-to-dhall Integer <<< 2 | dhall-to-json
json-to-dhall Natural <<< 2 | dhall-to-json
json-to-dhall Double <<< -2.345 | dhall-to-json
json-to-dhall Text <<< '"foo bar"' | dhall-to-json
json-to-dhall 'List Integer' <<< '[1, 2, 3]' | dhall-to-json
json-to-dhall '{foo : List Integer}' <<< '{"foo": [1, 2, 3]}' | dhall-to-json
json-to-dhall '{foo : List Integer}' <<< '{"foo": [1, 2, 3], "bar" : "asdf"}' | dhall-to-json
json-to-dhall '{ a : Integer, b : Text }' <<< '[{"key":"a", "value":1}, {"key":"b", "value":"asdf"}]' | dhall-to-json
json-to-dhall 'List { mapKey : Text, mapValue : Text }' <<< '{"foo": "bar"}' | dhall-to-json
json-to-dhall "Optional Integer" <<< '1' | dhall-to-json
json-to-dhall '{ a : Integer, b : Optional Text }' <<< '{ "a": 1 }' | dhall-to-json
json-to-dhall 'List < Left : Text | Right : Integer >' <<< '[1, "bar"]' | dhall-to-json
json-to-dhall '{foo : < Left : Text | Right : Integer >}' <<< '{ "foo": "bar" }' | dhall-to-json
json-to-dhall '{foo : < Left : Text | Middle : Text | Right : Integer >}' <<< '{ "foo": "bar"}' | dhall-to-json

json-to-dhall(x) without Dhall schema argument is generally not possible, obviously. One could have a separate type-guessing tool (JSON → Dhall Type) facilitating the schema authoring, but in many cases of practical importance when you basically need a tool for importing JSON data of a standard format over and over again (see spacchetti/spago for an example), manually preparing a Dhall schema for each format is a relatively minor one-time effort.

@Profpatsch
Copy link
Member

As long as the round-trip works it’s fine I guess.
I see, you have to give it a type separately, otherwise it couldn’t distinguish e.g. Double, Natural or Integer.

@antislava
Copy link
Collaborator Author

antislava commented Apr 2, 2019

Indeed! Also, you generally cannot derive Optional and Union from the JSON data (not from a single instantiation of a schema at least).

There are also situations where you would prefer to interpret/import a JSON object as a key-value association list rather than a Dhall record (for example, dependencies field in my bower.dhall).

So json-to-dhall(t, dhall-to-json(x : t)) ~ x is probably as close as we can get in terms of reversibility (without too many options/flags, constraints, and assumptions), but it is actually fine: t is basically our import configuration ;)

@Gabriella439
Copy link
Collaborator

This is great! I think the main thing we need is to upstream this into the standard, mainly to sort out corner cases (like how to handle unions, if at all), to document the expected behavior, and to give other implementations input into how JSON support evolves.

@antislava
Copy link
Collaborator Author

antislava commented Apr 3, 2019

@Gabriel439 I agree, the JSON interface needs to be standardized. The proper JSON importing functionality could increase Dhall usefulness and adoption significantly. Currently, in json-to-json, json-to-nix, etc... workflows, where the JSON source is provided by some external API (e.g. the GitHub API source) I need to resort to a pipe of jq (or maybe nix eval), probably some shell interpolation, and then jq or yq again, and adding Dhall in the middle currently doesn't improve the workflow enough to justify it.
Using Haskell script for importing and defining the Dhall type is straightforward but involves quite a bit of boilerplate and will (inevitably) lack consistent behavior and error reporting (and, more importantly, falling to Haskell kind of makes Dhall itself redundant, unless Dhall is not the script's target?). A standard Dhall.FromJSON module (with the conversion function, conversion parameters and the exception data type) would also greatly improve the Haskell approach.

Back to your suggestion, where should this upstream to more exactly (I am not yet very familiar with the project hierarchy). Is there a specific file or folder in dhall-lang/dhall-lang or dhall-lang/dhall-haskell?
So far, I just tried to follow the dhall-to-json's code as much as possible (but for now keeping all the logic in Main.hs - it will obviously have to be moved to something like src/Dhall/FromJSON.hs, if simultaneously renaming current Dhall.JSON to Dhall.ToJSON for consistency, and perhaps factoring out common parts to a separate shared module, named Dhall.JSON?). Is dhall-to-json already upstreamed in this sense so that I could piggyback on that as well?

@antislava
Copy link
Collaborator Author

antislava commented Apr 3, 2019

@Gabriel439 Regarding the corner cases, I added the following options/flags for importing Unions:

  1. --unions-first (default) selects the first successful match (error of none matched)
  2. --unions-strict results in 'undecidable union' error if there are more than one successful matches (all the alternatives being printed out in the error message)
  3. --unions-none forbids unions altogether (probably redundant since the user is responsible for the import schema anyway?)

With regards to the Records I added a --records-strict switch requiring all the keys in JSON object to be handled, reporting the error otherwise. The default behavior is to allow the unhanded keys (which I think is a more common situation as one normally needs a subset of an external API adding new fields incrementally if needed).

JSON key-value lists are imported as Records, while Dhall mapKey-mapValue lists match JSON objects (see bower.dhall for an example) by default. One can opt out of these defaults using --no-keyval-arrays and --no-keyval-maps switches, respectively.

Are there other important corner cases to consider? What did I miss?

@joneshf
Copy link
Collaborator

joneshf commented Apr 3, 2019

I think the main thing we need is to upstream this into the standard, mainly to sort out corner cases (like how to handle unions, if at all), to document the expected behavior, and to give other implementations input into how JSON support evolves.

Are you suggesting we standardize before merging this PR (and releasing the program) or are you saying we standardize as an auxiliary thing to do? I ask because as @antislava hinted to:

Is dhall-to-json already upstreamed in this sense so that I could piggyback on that as well?

There's no standardization for going the other way. And, printing seems way more important than parsing.

I'm all for standardization of JSON decoding/encoding. I don't think it's that big a burden to add, either. This isn't me fighting against standardizing. It's just a little unclear why we need it now before adding a program to a separate repo.

@f-f
Copy link
Member

f-f commented Apr 3, 2019

I agree with @joneshf that standardization should be an independent process from just merging this - we can always standardize the other way around (i.e. from here to the standard), as it happened with the main implementation.

I'll also add that I think implementing a JSON interface should not be compulsory for a given implementation (the usual reason is to lighten the implementation-bootstrapping process, as this feels a bit orthogonal)

@Gabriella439
Copy link
Collaborator

@antislava: I originally envisioned first upstreaming this into the standard, but I've changed my mind and think it's worth merging this even before we standardize it. If you open the pull request I'll accept it

@antislava
Copy link
Collaborator Author

antislava commented Apr 9, 2019

Done! #884

Note that I was getting some unrelated Dhall test failures after merging the latest master into my branch, so I had to disable tests in dhall-json's nix-shell temporarily (in nix/shared.nix - outside this commit obviously).

ps @Gabriel439 I keep seeing minor errors in Hydra builds (most recently, related to Semigroups/Monoids), which I don't see locally, using the "official" shell.nix and the cabal file. Please, advise how to align the local setup with that of the Hydra server, so that I could realistically iron out those. (Maybe, some advice on this should be included in the README, unless I missed it somehow?)

pps ah, I see it was ghc-7.10.3 build that was causing problems. I'd obviously need to build under import ./nix/shared.nix { compiler = "ghc7103"; }; locally to take care of those errors properly, but currently running out of space on my machine ;) Will try this during the weekend, disabling ghc < 8 for now in .cabal to see that everything else checks.

@singpolyma
Copy link
Contributor

For interest's sake: there are also experimental json-to-dhall and yaml-to-dhall tools shipped with dhall-ruby

@antislava
Copy link
Collaborator Author

antislava commented Apr 14, 2019

@singpolyma Thank you for the info! Could you please link to some list of simple examples of how it works in your repository? (Do you also require some sort of a schema as an input argument?)

@singpolyma
Copy link
Contributor

singpolyma commented Apr 14, 2019 via email

@Gabriella439
Copy link
Collaborator

I'll mark this resolved now that #884 is merged

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants