Converting Nomad Job Spec HCL to JSON and Back? #498
Comments
|
Hi @nickpoulos, The JSON variant of HCL relies on application-defined schema to resolve ambiguity inherent in the simpler JSON syntax (as compared to the HCL native syntax) so there is no general answer to "convert between native syntax and JSON syntax". Instead, this will always need to be an application-specific implementation that is aware of the application's own schema. There are some third-party tools (which you've found) which try to do this generically, in spite of the caveats, and they can broadly work as long as the application is using a schema that matches each tool's assumptions about what a "generic schema" might look like. Even then, the conversion is inherently one-way -- native syntax to JSON -- because the JSON syntax is "lossy" and not self-describing enough to convert to native syntax without reference to a schema. With all of that said then, I would agree with you that this seems like Nomad-specific format conversion seems like a lot of work to take on as a third-party individual. If you have a use-case for such a conversion then it may be better to share that use-case with the Nomad team and see if they would either address it by maintaining such a conversion tool themselves or by devising some other solution to meet your underlying problem. As things exist today though, I believe Nomad accepts both the native syntax and JSON syntax directly, with the assumption that you'll just use whichever one makes sense for each particular case. I work mainly on Terraform so I'm not sure if there are Nomad-specific constraints here that make that tougher, but even so I think it would be better to start this discussion in the Nomad repository so it can be frame in terms of your Nomad use-case rather than in terms of potential technical solutions. |
|
Would it be possible to do a functional 1:1 mapping if the export to/import from JSON included schema/annotation information that facilitated the conversion? If I understand correctly, this would require some reserved keywords that couldn't be present in any HCL, and it would require that, at a minimum, either applications that support "annotated" HCL-JSON be able to ignore these, or they be pruned prior to use? I think there are two separate problems that get tangled up in this.
If we treat these problems as one problem, it's easy to punt and say that converters must be able to satisfy the constraints of the application consuming HCL; after all, the HCL spec will never be able to know all schemas in advance. However, I think targeting just the first problem is both feasible and beneficial, and we can leave validation of specific schemas up to each application. Should I write this up as a separate issue? |
|
Hi @flurie, As we currently stand, there really is no such thing as "generic HCL". HCL is defined as a set of building blocks that languages can be defined in terms of, not as a self-contained language it its own right. I think you are proposing to create a third HCL syntax that is built from JSON grammar but intentionally more verbose in order to describe at least as much structure as the native syntax can describe without any ambiguity about what the author intends a particular construct to mean. I would agree that is possible in principle but it sounds like a lot of work and would necessarily create something that isn't compatible with today's HCL JSON, and so applications would need some way to determine which JSON syntax the configuration author intends to use. I think what I find most challenging about this issue is that it seems to be a classic "XY Problem". The problem statement is "I want to write some tools and so I need to work with HCL in JSON", but that doesn't really follow: the expected way to build tools is to interact with the HCL API to extract the information needed; converting to JSON or back should never be needed. But since this issue doesn't identify exactly what use-case this tool or tools would address, it's hard to suggest how to solve it using HCL as it exists or to identify specific new API features that would help address it within the existing abstraction. If there's some specific transformation or analysis to be done here then I'd prefer to talk about that, rather than talk about trying to achieve it by converting between syntaxes, which will always be lossy and therefore inappropriate for any tool which modifies configuration written by human authors. |
Did you know that Nomad has not 1 but 2 JSON formats for jobs? 2½ if you want to acknowledge that sometimes our JSON job representations have a Job top-level wrapper and sometimes do not. The 2½ formats are: ``` 1. HCL JSON 2. Input API JSON (top-level Job field) 2.5. Output API JSON (lacks top-level Job field) ``` `#2` is what our docs consider our API JSON. `#2.5` seems to be an accident of history we can't fix with breaking API compatibility. `#1` is an even more interesting accident of history: the `jobspec2` package automatically detects if the input to Parse is JSON and switches to a JSON parser. This behavior is undocumented, the format is unspecified, and there is no official HashiCorp tooling to produce this JSON from HCL. The plot thickens when you discover popular third party tools like hcl2json.com and https://github.com/tmccombs/hcl2json seem to produce JSON that `nomad run` accepts! Since we have no telemetry around whether or not anyone passes HCL JSON to `nomad run`, and people don't file bugs around features that Just Work, I'm choosing to leave that code path in place and *acknowledged but not suggested* in documentation. See hashicorp/hcl#498 for a more comprehensive discussion of what officially supporting HCL JSON in Nomad would look like. (I also added some of the missing fields to the (Input API flavor) JSON Job documentation, but it still needs a lot of work to be comprehensive.)
* docs: update json jobs docs Did you know that Nomad has not 1 but 2 JSON formats for jobs? 2½ if you want to acknowledge that sometimes our JSON job representations have a Job top-level wrapper and sometimes do not. The 2½ formats are: ``` 1. HCL JSON 2. Input API JSON (top-level Job field) 2.5. Output API JSON (lacks top-level Job field) ``` `#2` is what our docs consider our API JSON. `#2.5` seems to be an accident of history we can't fix with breaking API compatibility. `#1` is an even more interesting accident of history: the `jobspec2` package automatically detects if the input to Parse is JSON and switches to a JSON parser. This behavior is undocumented, the format is unspecified, and there is no official HashiCorp tooling to produce this JSON from HCL. The plot thickens when you discover popular third party tools like hcl2json.com and https://github.com/tmccombs/hcl2json seem to produce JSON that `nomad run` accepts! Since we have no telemetry around whether or not anyone passes HCL JSON to `nomad run`, and people don't file bugs around features that Just Work, I'm choosing to leave that code path in place and *acknowledged but not suggested* in documentation. See hashicorp/hcl#498 for a more comprehensive discussion of what officially supporting HCL JSON in Nomad would look like. (I also added some of the missing fields to the (Input API flavor) JSON Job documentation, but it still needs a lot of work to be comprehensive.) Co-authored-by: Tim Gross <tgross@hashicorp.com>
|
On the Nomad side this is still unresolved, but now somewhat intentionally instead of just by pure accident. My solution was not to resolve the I also added a That It's not a great state of affairs, but unless somebody has a strong reason to use HCL-JSON over the API-JSON format I don't think the Nomad team will invest time in trying to improve our HCL-JSON story. Sorry! I know it would have been nice for there to be One True JSON format for Nomad jobspecs, but I don't think we can do that without just dropping the existing informal HCL-JSON as removing API-JSON isn't an option. |
* docs: update json jobs docs Did you know that Nomad has not 1 but 2 JSON formats for jobs? 2½ if you want to acknowledge that sometimes our JSON job representations have a Job top-level wrapper and sometimes do not. The 2½ formats are: ``` 1. HCL JSON 2. Input API JSON (top-level Job field) 2.5. Output API JSON (lacks top-level Job field) ``` `#2` is what our docs consider our API JSON. `#2.5` seems to be an accident of history we can't fix with breaking API compatibility. `#1` is an even more interesting accident of history: the `jobspec2` package automatically detects if the input to Parse is JSON and switches to a JSON parser. This behavior is undocumented, the format is unspecified, and there is no official HashiCorp tooling to produce this JSON from HCL. The plot thickens when you discover popular third party tools like hcl2json.com and https://github.com/tmccombs/hcl2json seem to produce JSON that `nomad run` accepts! Since we have no telemetry around whether or not anyone passes HCL JSON to `nomad run`, and people don't file bugs around features that Just Work, I'm choosing to leave that code path in place and *acknowledged but not suggested* in documentation. See hashicorp/hcl#498 for a more comprehensive discussion of what officially supporting HCL JSON in Nomad would look like. (I also added some of the missing fields to the (Input API flavor) JSON Job documentation, but it still needs a lot of work to be comprehensive.) Co-authored-by: Tim Gross <tgross@hashicorp.com>
* docs: update json jobs docs Did you know that Nomad has not 1 but 2 JSON formats for jobs? 2½ if you want to acknowledge that sometimes our JSON job representations have a Job top-level wrapper and sometimes do not. The 2½ formats are: ``` 1. HCL JSON 2. Input API JSON (top-level Job field) 2.5. Output API JSON (lacks top-level Job field) ``` `#2` is what our docs consider our API JSON. `#2.5` seems to be an accident of history we can't fix with breaking API compatibility. `#1` is an even more interesting accident of history: the `jobspec2` package automatically detects if the input to Parse is JSON and switches to a JSON parser. This behavior is undocumented, the format is unspecified, and there is no official HashiCorp tooling to produce this JSON from HCL. The plot thickens when you discover popular third party tools like hcl2json.com and https://github.com/tmccombs/hcl2json seem to produce JSON that `nomad run` accepts! Since we have no telemetry around whether or not anyone passes HCL JSON to `nomad run`, and people don't file bugs around features that Just Work, I'm choosing to leave that code path in place and *acknowledged but not suggested* in documentation. See hashicorp/hcl#498 for a more comprehensive discussion of what officially supporting HCL JSON in Nomad would look like. (I also added some of the missing fields to the (Input API flavor) JSON Job documentation, but it still needs a lot of work to be comprehensive.) Co-authored-by: Tim Gross <tgross@hashicorp.com>
* no-op commit due to failed cherry-picking * docs: update json jobs docs (#12766) * docs: update json jobs docs Did you know that Nomad has not 1 but 2 JSON formats for jobs? 2½ if you want to acknowledge that sometimes our JSON job representations have a Job top-level wrapper and sometimes do not. The 2½ formats are: ``` 1. HCL JSON 2. Input API JSON (top-level Job field) 2.5. Output API JSON (lacks top-level Job field) ``` `#2` is what our docs consider our API JSON. `#2.5` seems to be an accident of history we can't fix with breaking API compatibility. `#1` is an even more interesting accident of history: the `jobspec2` package automatically detects if the input to Parse is JSON and switches to a JSON parser. This behavior is undocumented, the format is unspecified, and there is no official HashiCorp tooling to produce this JSON from HCL. The plot thickens when you discover popular third party tools like hcl2json.com and https://github.com/tmccombs/hcl2json seem to produce JSON that `nomad run` accepts! Since we have no telemetry around whether or not anyone passes HCL JSON to `nomad run`, and people don't file bugs around features that Just Work, I'm choosing to leave that code path in place and *acknowledged but not suggested* in documentation. See hashicorp/hcl#498 for a more comprehensive discussion of what officially supporting HCL JSON in Nomad would look like. (I also added some of the missing fields to the (Input API flavor) JSON Job documentation, but it still needs a lot of work to be comprehensive.) Co-authored-by: Tim Gross <tgross@hashicorp.com> Co-authored-by: temp <temp@hashicorp.com> Co-authored-by: Michael Schurter <mschurter@hashicorp.com> Co-authored-by: Tim Gross <tgross@hashicorp.com>
* docs: update json jobs docs Did you know that Nomad has not 1 but 2 JSON formats for jobs? 2½ if you want to acknowledge that sometimes our JSON job representations have a Job top-level wrapper and sometimes do not. The 2½ formats are: ``` 1. HCL JSON 2. Input API JSON (top-level Job field) 2.5. Output API JSON (lacks top-level Job field) ``` `#2` is what our docs consider our API JSON. `#2.5` seems to be an accident of history we can't fix with breaking API compatibility. `#1` is an even more interesting accident of history: the `jobspec2` package automatically detects if the input to Parse is JSON and switches to a JSON parser. This behavior is undocumented, the format is unspecified, and there is no official HashiCorp tooling to produce this JSON from HCL. The plot thickens when you discover popular third party tools like hcl2json.com and https://github.com/tmccombs/hcl2json seem to produce JSON that `nomad run` accepts! Since we have no telemetry around whether or not anyone passes HCL JSON to `nomad run`, and people don't file bugs around features that Just Work, I'm choosing to leave that code path in place and *acknowledged but not suggested* in documentation. See hashicorp/hcl#498 for a more comprehensive discussion of what officially supporting HCL JSON in Nomad would look like. (I also added some of the missing fields to the (Input API flavor) JSON Job documentation, but it still needs a lot of work to be comprehensive.) Co-authored-by: Tim Gross <tgross@hashicorp.com>
* docs: update json jobs docs Did you know that Nomad has not 1 but 2 JSON formats for jobs? 2½ if you want to acknowledge that sometimes our JSON job representations have a Job top-level wrapper and sometimes do not. The 2½ formats are: ``` 1. HCL JSON 2. Input API JSON (top-level Job field) 2.5. Output API JSON (lacks top-level Job field) ``` `#2` is what our docs consider our API JSON. `#2.5` seems to be an accident of history we can't fix with breaking API compatibility. `#1` is an even more interesting accident of history: the `jobspec2` package automatically detects if the input to Parse is JSON and switches to a JSON parser. This behavior is undocumented, the format is unspecified, and there is no official HashiCorp tooling to produce this JSON from HCL. The plot thickens when you discover popular third party tools like hcl2json.com and https://github.com/tmccombs/hcl2json seem to produce JSON that `nomad run` accepts! Since we have no telemetry around whether or not anyone passes HCL JSON to `nomad run`, and people don't file bugs around features that Just Work, I'm choosing to leave that code path in place and *acknowledged but not suggested* in documentation. See hashicorp/hcl#498 for a more comprehensive discussion of what officially supporting HCL JSON in Nomad would look like. (I also added some of the missing fields to the (Input API flavor) JSON Job documentation, but it still needs a lot of work to be comprehensive.) Co-authored-by: Tim Gross <tgross@hashicorp.com> Co-authored-by: temp <temp@hashicorp.com> Co-authored-by: Tim Gross <tgross@hashicorp.com>
It has been very difficult to prop up some basic tooling around HCL based config files.
There are multiple projects that try to help with this issue but I have not been able to find one to do this correctly for my use case. I am hoping somebody might be able to point me in the right direction.
I have tried:
1. Using the
nomadcli.This would be the ideal way except for a few things.
It wants to know the values of vars ahead of time so it can interpolate them. I need to keep the variable names and ${} syntax in place.
There seems to be no way to go back to HCL once we have the JSON
2. Using a tool like https://github.com/hashicorp/hcl/tree/main/cmd/hcldec
This would also be ideal, but one big exception:
3. Using third party tools like https://github.com/tmccombs/hcl2json.
This particular ones seems to work well...almost.
It leaves my variables and their related syntax alone so that is a +1.
It does not quite generate the right JSON format for Nomad specs. Ex. Constraints, Templates (which are key for me)
There is no way to go back to HCL after
4. Another third party tool - https://github.com/kvz/json2hcl.
After digging into the code a little I found https://github.com/hashicorp/nomad/blob/main/jobspec/parse_job.go but could not find where it runs or how I might be able to use it.
Does anybody have any advice? Or has taken a stab at writing a Nomad
hcldecspec file? Doeshcldecdo the reverse conversion?Thanks
The text was updated successfully, but these errors were encountered: